Large multimodal models (LMMs) typically process visual inputs with uniform resolution across the entire field of view, leading to inefficiencies when non-critical image regions are processed as precisely as key areas. Inspired by the human visual system’s foveated approach, we apply a sampling method to leading architectures such as MDETR, BLIP2, InstructBLIP, LLaVA, and ViLT, and evaluate their performance with variable (foveated) resolution inputs. Results show that foveated sampling boosts accuracy in visual tasks like question answering and object detection under tight pixel budgets, improving performance by up to 2.7% on the GQA dataset, 2.1% on SEED-Bench, and 2.0% on VQAv2 compared to uniform sampling. Furthermore, we show that indiscriminate resolution increases yield diminishing returns, with models achieving up to 80% of their full capability using just 3% of the pixels, even on complex tasks. Foveated sampling prompts more human-like processing within models, such as neuronal selectivity and globally acting self-attention in vision transformers. This paper provides a foundational analysis of foveated sampling’s impact on existing models, suggesting that more efficient architectural adaptations, mimicking human visual processing, are a promising research venue for the community. Potential applications of our findings center low power minimal bandwidth devices (such as UAVs and edge devices), where compact and efficient vision is critical.

In this paper, we explore the capability of an agent to construct a logical sequence of action steps, thereby as-sembling a strategic procedural plan. This plan is cru-cial for navigating from an initial visual observation to a target visual outcome, as depicted in real-life instructional videos. Existing works have attained partial suc-cess by extensively leveraging various sources of information available in the datasets, such as heavy intermediate visual observations, procedural names, or natural language step-by-step instructions, for features or supervision signals. However, the task remains formidable due to the implicit causal constraints in the sequencing of steps and the variability inherent in multiple feasible plans. To tackle these intricacies that previous efforts have over-looked, we propose to enhance the agent's capabilities by infusing it with procedural knowledge. This knowledge, sourced from training procedure plans and structured as a directed weighted graph, equips the agent to better nav-igate the complexities of step sequencing and its poten-tial variations. We coin our approach KEPP, a novel Knowledge-Enhanced Procedure Planning system, which harnesses a probabilistic procedural knowledge graph ex-tracted from training data, effectively acting as a compre-hensive textbook for the training domain. Experimental evaluations across three widely-used datasets under set-tings of varying complexity reveal that KEPP attains su-perior, state-of-the-art results while requiring only mini-mal supervision. Code and trained model are available at https://github.com/Ravindu-Yasas-Nagasinghe/KEPP

Understanding language goes hand in hand with the ability to integrate complex contextual information obtained via perception.In this work, we present a novel task for grounded language understanding: disambiguating a sentence given a visual scene which depicts one of the possible interpretations of that sentence.To this end, we introduce a new multimodal corpus containing ambiguous sentences, representing a wide range of syntactic, semantic and discourse ambiguities, coupled with videos that visualize the different interpretations for each sentence.We address this task by extending a vision model which determines if a sentence is depicted by a video.We demonstrate how such a model can be adjusted to recognize different interpretations of the same underlying sentence, allowing to disambiguate sentences in a unified fashion across the different ambiguity types.

Detecting an object part relies on two sources of information - the appearance of the part itself and the context supplied by surrounding parts. In this paper we consider problems in which a target part cannot be recognized reliably using its own appearance, such as detecting low-resolution hands, and must be recognized using the context of surrounding parts. We develop the `chains model' which can locate parts of interest in a robust and precise manner, even when the surrounding context is highly variable and deformable. In the proposed model, the relation between context features and the target part is modeled in a non-parametric manner using an ensemble of feature chains leading from parts in the context to the detection target. The method uses the configuration of the features in the image directly rather than through fitting an articulated 3-D model of the object. In addition, the chains are composable, meaning that new chains observed in the test image can be composed of sub-chains seen during training. Consequently, the model is capable of handling object poses which are infrequent, even non-existent, during training. We test the approach in different settings, including object parts detection, as well as complete object detection. The results show the advantages of the chains model for detecting and localizing parts of complex deformable objects.

A computer-based method for the assessment of body-image distortions in anorexia nervosa and other eating-disorder patients is presented in this paper. At the core of the method is a realistic pictorial simulation of lifelike weight changes, applied to a real source image of the patient. The patients, using a graphical user interface, adjust their body shapes until they meet their self-perceived appearance. Measuring the extent of virtual fattening or slimming of a body with respect to its real shape and size allows direct quantitative evaluation of the cognitive distortion in body image. In a preliminary experiment involving 33 anorexia-nervosa patients, 70% of the subjects chose an image with simulated visual weight gain between 8%-16% as their "real" body image, while only one of them recognized the original body image. In a second experiment involving 30 healthy participants, the quality of the weight modified images was evaluated by pairwise selection trials. Over a weight change range from -16% to +28%, in about 30% of the trials, artificially modified images were mistakenly taken as "original" images, thus demonstrating the quality of the artificial images. The method presented is currently in a clinical validation phase, toward application in the research, diagnosis, evaluation, and treatment of eating disorders.