When a vision model performs image recognition, which visual attributes drive its predictions? Detecting unintended reliance on specific visual features is critical for ensuring model robustness, preventing overfitting, and avoiding spurious correlations. We introduce an automated framework for detecting such dependencies in trained vision models. At the core of our method is a self-reflective agent that systematically generates and tests hypotheses about visual attributes that a model may rely on. This process is iterative: the agent refines its hypotheses based on experimental outcomes and uses a self-evaluation protocol to assess whether its findings accurately explain model behavior. When inconsistencies arise, the agent self-reflects over its findings and triggers a new cycle of experimentation. We evaluate our approach on a novel benchmark of 130 models designed to exhibit diverse visual attribute dependencies across 18 categories. Our results show that the agent's performance consistently improves with self-reflection, with a significant performance increase over non-reflective baselines. We further demonstrate that the agent identifies real-world visual attribute dependencies in state-of-the-art models, including CLIP's vision encoder and the YOLOv8 object detector.

Generating long and semantic-coherent reports to describe medical images poses great challenges towards bridging visual and linguistic modalities, incorporating medical domain knowledge, and generating realistic and accurate descriptions. We propose a novel Knowledge-driven Encode, Retrieve, Paraphrase (KERP) approach which reconciles traditional knowledge- and retrieval-based methods with modern learning-based methods for accurate and robust medical report generation. Specifically, KERP decomposes medical report generation into explicit medical abnormality graph learning and subsequent natural language modeling. KERP first employs an Encode module that transforms visual features into a structured abnormality graph by incorporating prior medical knowledge; then a Retrieve module that retrieves text templates based on the detected abnormalities; and lastly, a Paraphrase module that rewrites the templates according to specific cases. The core of KERP is a proposed generic implementation unit—Graph Transformer (GTR) that dynamically transforms high-level semantics between graph-structured data of multiple domains such as knowledge graphs, images and sequences. Experiments show that the proposed approach generates structured and robust reports supported with accurate abnormality description and explainable attentive regions, achieving the state-of-the-art results on two medical report benchmarks, with the best medical abnormality and disease classification accuracy and improved human evaluation performance.

Accurate and robust facial landmark tracking is a crucial step for face recognition and affect analysis systems. We often want to not only detect facial landmarks in images but to be able to track them reliably and consistently over time. Recently there has been an increase in research interest in facial landmark detection, especially in cascaded regression based methods such as the Supervised Descent Method (SDM). However, while facial landmark detection in images has improved significantly, comparably very little attention has been given to the task of landmark detection/tracking in videos. In our work we present a novel initialization procedure that can help with cascaded regression based facial landmark detection and tracking. Our initialization technique exploits the fact that cascaded regression is sensitive to initialization noise, especially in the presence of out-of-plane head pose variation, e.g. when a person is looking down when reading or during fast head motion. Our approach allows to learn good candidates for initialization, that we exploit in our tracking framework. We evaluate our technique on 300VW dataset – a large publicly available corpus of in-the-wild videos and demonstrate its effectiveness for a number of cascaded-regression landmark detection approaches.

We consider the problem of communicating data from energy-constrained distributed sensors. To reduce energy requirements, we go beyond the source reconstruction problem classically addressed, and focus on the problem where the recipient wants to perform supervised learning and classification on the data received from the sensors. Restricting our attention to a noiseless communication setting under simplistic Gaussian source assumptions, we study supervised learning and classification under total energy limitations. The energy constraints are modeled in two ways: 1) a linear scaling and 2) an exponential scaling of energy with number of bits used for compression at sensors. We first assume that the underlying parameters for Gaussian distributions have already been learned, and obtain (with linear scaling, reverse-waterfilling-type) strategies for allocating energy, and thus, bits, across different sensors under these two models. Intuitively, these strategies allocate larger rates and energies to sensors that are more “relevant” for the classification goal. These strategies are used to obtain an achievable bound on the tradeoff between energy and error-probability (classification risk). We then provide an algorithm for learning the distribution-parameters of the sensor-data under energy constraints to arrive at high-reliability energy-allocation strategies, while enabling the energy-allocation algorithm to backtrack when the underlying distributions change, or when there is noise in sensed data that can push the algorithm toward a local minimum. Finally, we provide numerical results on energy-savings for classification of simulated data as well as neural data acquired from electrocorticography (ECoG) experiments.