Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, Yejin Choi. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

The commonsense reasoning capabilities of vision-language models (VLMs), especially in abductive reasoning and defeasible reasoning, remain poorly understood. Most benchmarks focus on typical visual scenarios [1], [23], [42], making it difficult to discern whether model performance stems from keen perception and reasoning skills, or reliance on pure statistical recall. We argue that by focusing on atypical events in videos, clearer insights can be gained on the core capabilities of VLMs. Explaining and understanding such out-of-distribution events requires models to extend beyond basic pattern recognition and regurgitation of their prior knowledge. To this end, we introduce Black-SwanSuite, a benchmark for evaluating VLMs’ ability to reason about unexpected events through abductive and defeasible tasks. Our tasks artificially limit the amount of visual information provided to models while questioning them about hidden unexpected events, or provide new visual information that could change an existing hypothesis about the event. We curate a comprehensive benchmark suite comprising over 3,800 MCQ, 4,900 generative and 6,700 yes/no questions, spanning 1,655 videos. After extensively evaluating various state-of-the-art VLMs, including GPT-4o and Gemini 1.5 Pro, as well as open-source VLMs such as LLaVA-Video, we find significant performance gaps of up to 32% from humans on these tasks. Our findings reveal key limitations in current VLMs, emphasizing the need for enhanced model architectures and training strategies. Our data and leaderboard is available at https://blackswan.cs.ubc.ca.

An embodied AI assistant operating on egocentric video must integrate spatial cues across time - for instance, determining where an object A, glimpsed a few moments ago lies relative to an object B encountered later. We introduce Disjoint-3DQA , a generative QA benchmark that evaluates this ability of VLMs by posing questions about object pairs that are not co-visible in the same frame. We evaluated seven state-of-the-art VLMs and found that models lag behind human performance by 28%, with steeper declines in accuracy (60% → 30 %) as the temporal gap widens. Our analysis further reveals that providing trajectories or bird’s-eye-view projections to VLMs results in only marginal improvements, whereas providing oracle 3D coordinates leads to a substantial 20% performance increase. This highlights a core bottleneck of multi-frame VLMs in constructing and maintaining 3D scene representations over time from visual signals. Disjoint-3DQA therefore sets a clear, measurable challenge for long-horizon spatial reasoning and aims to catalyze future research at the intersection of vision, language, and embodied AI.

Sahithya Ravi, Patrick Huber, Akshat Shrivastava, Vered Shwartz, Arash Einolghozati. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Despite recent advancements in visionlanguage models, their performance remains suboptimal on images from non-western cultures, due to underrepresentation in training datasets.Various benchmarks have been proposed to test models' cultural inclusivity, but they have limited coverage of cultures and do not adequately assess cultural diversity across universal as well as culture-specific local concepts.To address these limitations, we introduce the GLOBALRG benchmark, comprising two challenging tasks: retrieval across universals and cultural visual grounding.The former task entails retrieving culturally-diverse images for universal concepts from 50 countries, while the latter aims at grounding culture-specific concepts within images from 15 countries.Our evaluation across a wide range of models reveals that the performance varies significantly across cultures -underscoring the necessity for enhancing multicultural understanding in vision-language models.Our

Event coreference models cluster event mentions pertaining to the same real-world event.Recent models rely on contextualized representations to recognize coreference among lexically or contextually similar mentions.However, models typically fail to leverage commonsense inferences, which is particularly limiting for resolving lexically-divergent mentions.We propose a model that extends event mentions with temporal commonsense inferences.Given a complex sentence with multiple events, e.g., "The man killed his wife and got arrested", with the target event "arrested", our model generates plausible events that happen before the target event -such as "the police arrived", and after it, such as "he was sentenced".We show that incorporating such inferences into an existing event coreference model improves its performance, and we analyze the coreferences in which such temporal knowledge is required. ① Mention Detection② Baseline Pairwise Scorer (spent, recovering): 0.14 (spent, gunshots): 0.05 (spent, shot): 0.1 (gunshots, shot): 0.82 … gunshots, shot, … recovering hospitalized spent Document 2The third coworker, Bryant Dalton, 39, spent two weeks in the hospital and still is recovering from gunshots to the neck and shoulder, prosecutors said. Document 1Bryant Dalton, 39, was shot in the neck and is hospitalized in good condition.… Document N … (spent, recovering): 0.14 (spent, gunshots): 0.05 (spent, shot): 0.1 (gunshots, shot): 0.82 … ③ Agglomerative Clustering gunshots, shot, … recovering spent, hospitalized spent, recovering, gunshots, shot, hospitalized ② Our Pairwise Scorer (spent, hospitalized): 0.1 (spent, hospitalized): 0.75 f(ctx spent , ctx hospitalized ) f(ctx spent , ctx hospitalized , cs spent , cs hospitalized ,)

There has been a growing interest in solving Visual Question Answering (VQA) tasks that require the model to reason beyond the content present in the image. In this work, we focus on questions that require commonsense reasoning. In contrast to previous methods which inject knowledge from static knowledge bases, we investigate the incorporation of contextualized knowledge using Commonsense Transformer (COMET), an existing knowledge model trained on human-curated knowledge bases. We propose a method to generate, select, and encode external commonsense knowledge alongside visual and textual cues in a new pre-trained Vision-Language-Commonsense transformer model, VLC-BERT. Through our evaluation on the knowledge-intensive OK-VQA and A-OKVQA datasets, we show that VLC-BERT is capable of outperforming existing models that utilize static knowledge bases. Furthermore, through a detailed analysis, we explain which questions benefit, and which don’t, from contextualized commonsense knowledge from COMET. Code: https://github.com/aditya10/VLC-BERT

OpenML is an online platform for open science collaboration in machine learning, used to share datasets and results of machine learning experiments. In this paper, we introduce OpenML-Python, a client API for Python, which opens up the OpenML platform for a wide range of Python-based machine learning tools. It provides easy access to all datasets, tasks and experiments on OpenML from within Python. It also provides functionality to conduct machine learning experiments, upload the results to OpenML, and reproduce results which are stored on OpenML. Furthermore, it comes with a scikit-learn extension and an extension mechanism to easily integrate other machine learning libraries written in Python into the OpenML ecosystem. Source code and documentation are available at https://github.com/openml/openml-python/.

Narrowband Internet of Things (NB-IoT) is a new Low Power Wide Area Network (LPWAN) technology released by the Third Generation Partnership Project (3GPP). The primary goals of NB-IoT are enhanced coverage, low cost, and long battery life. In order to enhance coverage, NB-IoT has new features, such as increasing transmission repetitions, decreasing bandwidth, and adapting the Modulation and Coding Scheme (MCS). In this paper, we present an implementation of these three features of NB-IoT in NS-3, an end-to-end network simulator. Using the developed simulation framework, the influence of the coverage enhancement features on network reliability and latency is evaluated. Furthermore, we propose a hybrid link adaptation strategy based on all three features, which tries to achieve optimal latency and coverage. To achieve this, we formulate and solve an optimization problem that finds the optimal value of repetitions, bandwidth and MCS such that the latency is minimum and the reliability is maintained. Based on the hybrid link adaption strategy, a new scheduler is implemented and evaluated in the NS-3 simulator. Through numerical results we show that the hybrid link adaptation method achieves lower latency and higher coverage than any of the coverage enhancement techniques. We also show that the proposed optimization method achieves the same performance as the exhaustive search method but with lower complexity.