Hengyuan Zhang, Shiping Yang, Xiao Liang, Chenming Shang, Yuxuan Jiang, Chaofan Tao, Jing Xiong, Hayden Kwok-Hay So, Ruobing Xie, Angel X. Chang, Ngai Wong. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Hengyuan Zhang, Chenming Shang, Sizhe Wang, Dongdong Zhang, Yiyao Yu, Feng Yao, Renliang Sun, Yujiu Yang, Furu Wei. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

The remarkable success of the autoregressive paradigm has made significant advancement in Multimodal Large Language Models (MLLMs), with powerful models like Show-o, Transfusion and Emu3 made notable strides in unified image understanding and generation. For the first time, we uncover a common phenomenon: the understanding capability of MLLMs is usually stronger than their generative capability, with a significant gap between them. Building on this insight, we propose HermesFlow, a simple and general framework designed to seamlessly bridge the gap between understanding and generation in MLLMs. Specifically, we take the homologous data as input to curate homologous preference data of both understanding and generation. Through Pair-DPO and self-play iterative optimization, HermesFlow effectively aligns multimodal understanding and generation using homologous preference data. Extensive experiments demonstrate the significant superiority of our approach over prior methods, particularly in narrowing the gap between multimodal understanding and generation. These findings highlight the potential of HermesFlow as a general alignment framework for next-generation multimodal foundation models.

Knowledge tracing (KT) plays a crucial role in predicting students’ future performance by analyzing their historical learning processes. Deep neural networks (DNNs) have shown great potential in solving the KT problem. However, there still exist some important challenges when applying deep learning techniques to model the KT process. The first challenge lies in modeling the individual question information. This is crucial because students’ knowledge acquisition on questions that share the same set of knowledge components (KCs) may vary significantly. However, due to the large question bank, the average number of interactions per question may not be sufficient. This limitation can potentially result in overfitting of the question embedding and inaccurate question knowledge acquisition state that relies on its corresponding question representation. Furthermore, there is a considerable portion of questions receiving relatively less interaction from students in comparison to the majority of questions. This can further increase the risk of overfitting and lower the accuracy of the obtained question knowledge acquisition state. The second challenge lies in interpreting the prediction results from existing deep learning-based KT models. In real-world applications, while it may not be necessary to have complete transparency and interpretability of the model parameters, it is crucial to present the model’s prediction results in a manner that teachers find interpretable. This makes teachers accept the rationale behind the prediction results and utilize them to design teaching activities and tailored learning strategies for students. However, the inherent black-box nature of deep learning techniques often poses a hurdle for teachers to fully embrace the model’s prediction results. To address these challenges, we propose a Question-centric Multi-experts Contrastive Learning framework for KT called Q-MCKT. This framework explicitly models students’ knowledge acquisition state at both the question and concept levels. It leverages the mixture of experts technique to capture a more robust and accurate knowledge acquisition state in both question and concept levels for prediction. Additionally, a fine-grained question-centric contrastive learning task is introduced to enhance the representations of less interactive questions and improve the accuracy of their corresponding question knowledge acquisition states. Moreover, Q-MCKT utilizes an item response theory-based prediction layer to generate interpretable prediction results based on the knowledge acquisition states obtained from the question and concept knowledge acquisition modules. We evaluate the proposed Q-MCKT framework on four public real-world educational datasets. The experimental results demonstrate that our approach outperforms a wide range of deep learning-based KT models in terms of prediction accuracy while maintaining better model interpretability. To ensure reproducibility, we have provided all the datasets and code on our website at https://github.com/rattlesnakey/Q-MCKT .

Concept Bottleneck Models (CBMs) map the black-box visual representations extracted by deep neural networks onto a set of interpretable concepts and use the concepts to make predictions, enhancing the transparency of the decision-making process. Multimodal pre-trained models can match visual representations with textual concept embeddings, allowing for obtaining the interpretable concept bottleneck without the expertise concept annotations. Recent research has focused on the concept bank establishment and the high-quality concept selection. However, it is challenging to construct a comprehensive concept bank through humans or large language models, which severely limits the performance of CBMs. In this work, we propose the Incremental Residual Concept Bottleneck Model (Res-CBM) to address the challenge of concept completeness. Specifically, the residual concept bottleneck model employs a set of optimizable vectors to complete missing concepts, then the incremental concept discovery module converts the complemented vectors with unclear meanings into potential concepts in the candidate concept bank. Our approach can be applied to any user-defined concept bank, as a post-hoc processing method to enhance the performance of any CBMs. Furthermore, to measure the descriptive efficiency of CBMs, the Concept Utilization Efficiency (CUE) metric is proposed. Experiments show that the Res-CBM outperforms the current state-of-the-art methods in terms of both accuracy and efficiency and achieves comparable performance to black-box models across multiple datasets.

Body Mass Index (BMI) is an important measurement of human obesity and health, which can provide useful information for plenty of practical purposes, such as monitoring, re-identification, and health care. Recently, some data-driven advances have been proposed to estimate BMI by 2D or 3D features from face images, frontal-body images and RGB-D images. However, due to the privacy issue or limitations of 3D cameras, the required data is hard to be obtained. More importantly, each of the previous works has only studied for a single type of features, hence it is worth investigating whether combinations of different features are more effective. To address this issue, we analyze the correlation of various features extracted from 2D body images with the estimated BMI, and then propose an accurate BMI estimation method with the optimal feature combination. Extensive experiments demonstrate that the proposed method outperforms these image-based BMI estimation methods which only utilize a single type feature in most cases. Code has been made available at : https://github.com/FVL2020/Features_for_BMI_estimation.