Contrastive Language-Image Pretraining (CLIP) enables zero-shot inference in downstream tasks such as image-text retrieval and classification.However, recent works extending CLIP suffer from the issue of modality gap, which arises when the image and text embeddings are projected to disparate manifolds, deviating from the intended objective of image-text contrastive learning.We discover that this phenomenon is linked to the modality-specific characteristic that each image or text encoder independently possesses.Herein, we propose two methods to address the modality gap: (1) a post-hoc embedding standardization method, I0T post that reduces the modality gap approximately to zero and (2) a trainable method, I0T async , to alleviate the modality gap problem by adding two normalization layers for each encoder.Our I0T framework can significantly reduce the modality gap while preserving the original embedding representations of trained models with their locked parameters.In practice, I0T post can serve as an alternative explainable automatic evaluation metric of widely used CLIPScore (CLIP-S).The code is available in https://github.com/xfactlab/I0T.

Zahra Bayramli, Ayhan Suleymanzade, Na Min An, Huzama Ahmad, Eunsu Kim, Junyeong Park, James Thorne, Alice Oh. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Wan Ju Kang, Eunki Kim, Na Min An, Sangryul Kim, Haemin Choi, Ki Hoon Kwak, James Thorne. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Image Captioning for state-of-the-art VLMs has significantly improved over time; however, this comes at the cost of increased computational complexity, making them less accessible for resource-constrained applications such as mobile devices and assistive technologies.Alternatively, comparably smaller VLMs prioritize high-level scene descriptions, overlooking finer details that contribute to a richer understanding of an image.In this paper, we introduce a training-free framework that enhances caption diversity and informativeness by explicitly attending to distinct image regions using a comparably small VLM, BLIP, as the backbone.Our approach leverages structured segmentation to produce hierarchical representations that capture both global and localized semantics.Without requiring additional model training, we demonstrate that our method allows smaller VLMs to achieve performance comparable to larger models in terms of image-caption alignment, semantic integrity, and diversity.We evaluate our framework on MSCOCO, Flickr30k, and Nocaps test datasets, achieving a Div-2 score of 0.735, 0.750, and 0.748 for each dataset, respectively, while maintaining strong image-caption relevancy and semantic integrity with the humanannotated captions.Our code is available at https://github.com/xfactlab/HBoP.

Stable pre-training is essential for achieving better-performing language models.However, tracking pre-training stability by calculating gradient variance at every step is impractical due to the significant computational costs.We explore Token Embedding Variability (TEV) as a simple and efficient proxy for assessing pre-training stability in language models with pre-layer normalization, given that shallower layers are more prone to gradient explosion (section 2.2).Moreover, we propose Multihead Low-Rank Attention (MLRA) as an architecture to alleviate such instability by limiting the exponential growth of output embedding variance, thereby preventing the gradient explosion (section 3.2).Empirical results on GPT-2 with MLRA demonstrate increased stability and lower perplexity, particularly in deeper models.

The quality of the artificial vision produced by visual prostheses has traditionally been evaluated by human psychophysical tests using images expressed with an array of phosphenes. However, such experiments involving human subjects are considerably time-consuming and labor-intensive. One potential solution may be implementing an efficient approach to assist or replace psychophysical experiments showing a short-term learning effect in human subjects. The present work developed a reinforcement learning (RL)-based feedback framework which built artificial agents to emulate the behavioral changes in the learning process of human subjects over the experimental trials. In our framework, we first trained an agent which can gradually learn to identify 720 faces presented in low-resolution phosphene images with feedback rewards received from the agreement in the perception of nine training human subjects. Then, in the automating stage of the framework, we created nine RL agents. By testing those agents, we found the RL agents mimicked the learning effects of nine test human subjects better than nine instances of a supervised learning (SL) model. Given the similar outcomes with human tests and the time efficiency of RL, our framework may expedite the development of visual prosthetic systems by at least partially replacing laborious human psychophysical experiments.