We address the problem of designing interpretable algorithms for image classification. Modern computer vision algorithms implement classification in two phases: feature extraction - the encoding - that relies on deep neural networks (DNN), followed by a task-oriented decision - the decoding - often also using a DNN. We propose to formulate this last phase as an argumentative DialoguE Between two agents relying on visual ATtributEs and Similarity to prototypes (DEBATES). DEBATES represents the combination of information provided by two encoders in a transparent and interpretable way. It relies on a dual process that combines similarity to prototypes and visual attributes, each extracted from an encoder. DEBATES makes explicit the agreements and conflicts between the two encoders managed by the two agents, reveals the causes of unintended behaviors, and helps identify potential corrective actions to improve performance. The approach is demonstrated on two problems of fine-grained image classification.

Adversarial patch attacks, where a small patch is placed in the scene to fool neural networks, have been studied for numerous applications. Focusing on image classification, we consider the setting of a black-box transfer attack where an attacker does not know the target model. Instead of forcing corrupted image representations to cross the nearest decision boundaries or converge to a particular point, we propose a distribution-oriented approach. We rely on optimal transport to push the feature distribution of attacked images towards an already modeled distribution. We show that this new distribution-oriented approach leads to better transferable patches. Through digital experiments conducted on ImageNet-1K, we provide evidence that our new patches are the only ones that can simultaneously influence multiple Transformer models and Convolutional Neural Networks. Physical world experiments demonstrate that our patch can affect systems in deployment without explicit knowledge.

Vision-Language Models (VLMs) trained on millions of images excel at generalization in various domains. However, a significant distribution shift between remote sensing and natural images limits their effectiveness without adaptation. We investigate both the ability of VLMs to adapt to satellite images and the ability of previously adapted VLMs to detect new classes of objects on these images with only few examples. We address this challenge by exploring different parameter-efficient finetuning strategies to adapt to remote sensing the knowledge acquired by these models during their large pre-training. We evaluate these strategies in few-shot settings and scenarios involving training without data constraints. We also show on the DIOR dataset that a GLIP model previously trained on satellite images is able to achieve state-of-the-art performance in the few-shot detection of new classes of objects, without requiring a dedicated training strategy.

Continual learning in dynamic environments is a challenge for large-scale machine learning models. This research addresses Domain Incremental Learning (DIL), a setting where the goal is to incrementally increase the input data scope of a model. More specifically, we investigate the possibility of using foundation models (FMs) as a fixed feature extractor combined with a PPCA that can be sequentially and accurately updated. Focusing on the classification of VHR remote sensing (RS) images, we show on the FLAIR#1 dataset that this simple DIL strategy achieves competitive accuracy compared to memory-based baselines across different pre-trained sources. We also compare different types of foundation models and highlight the importance of data diversity over data specialization to improve the quality of FMs.

For classification models based on neural networks, the maximum predicted class probability is often used as a confidence score. This score rarely predicts well the probability of making a correct prediction and requires a post-processing calibration step. However, many confidence calibration methods fail for problems with many classes. To address this issue, we transform the problem of calibrating a multiclass classifier into calibrating a single surrogate binary classifier. This approach allows for more efficient use of standard calibration methods. We evaluate our approach on numerous neural networks used for image or text classification and show that it significantly enhances existing calibration methods.

This paper makes a contribution to the problem of incremental class learning, the principle of which is to sequentially introduce batches of samples annotated with new classes during the learning phase. The main objective is to reduce the drop in classification performance on old classes, a phenomenon commonly called catastrophic forgetting. We propose in this paper a new method which exploits the availability of a large quantity of non-annotated images in addition to the annotated batches. These images are used to regularize the classifier and give the feature space a more stable structure. We demonstrate on two image data sets, MNIST and STL-10, that our approach is able to improve the global performance of classifiers learned using an incremental learning protocol, even with annotated batches of small size.

This paper presents an exemplar based metric learning framework dedicated to robust visual localization in complex scenes, e.g. street images. The proposed framework learns off-line a specific (local) metric for each image of the database, so that the distance between a database image and a query image representing the same scene is smaller than the distance between the current image and other images of the database. To achieve this goal, we generate geometric and photometric transformations as proxies for query images. From the generated constraints, the learning problem is cast as a convex optimization problem over the cone of positive semi-definite matrices, which is efficiently solved using a projected gradient descent scheme. Successful experiments, conducted using a freely available geo-referenced image database, reveal that the proposed method significantly improves results over the metric in the input space, while being as efficient at test time. In addition, we show that the model learns discriminating features for the localization task, and is able to gain invariance to meaningful transformations.

In this paper, we address the task of semantic labeling of multisource earth-observation (EO) data. Precisely, we benchmark several concurrent methods of the last 15 years, from expert classifiers, spectral support-vector classification and high-level features to deep neural networks. We establish that (1) combining multisensor features is essential for retrieving some specific classes, (2) in the image domain, deep convolutional networks obtain significantly better overall performances and (3) transfer of learning from large generic-purpose image sets is highly effective to build EO data classifiers.

Aerial video sequences are a common source for applications such as intelligence, surveillance or search and rescue. Their off-line analysis however requires a certain level of assistance to reduce the expert's workload. This study focuses on detecting mobile vehicles in such sequences. The proposed approach exploits two types of contextual information: loose user input as tagged areas in a reference frame, and knowledge-based priors to describe specific constraints. Our main contribution is the design of a two-step general framework able to combine these two types of information. The first step is a pixelwise semantic classification labelling each sequence frame structure in vehicle, road and background; the classifier is based on local motion and appearance features and is organized as an iterative refining process. The second step exploits knowledge-based spatial reasoning to filter out false alarms. A quantitative evaluation on real video sequences demonstrates the usefulness of each level of contextual information.

This paper describes a method for detecting salient regions in remote-sensed images, based on scale and contrast interaction. We consider the focus on salient structures as the first stage of an object detection/recognition algorithm, where the salient regions are those likely to contain objects of interest. Salient objects are modeled as spatially localized and contrasted structures with any kind of shape or size. Their detection exploits a probabilistic mixture model that takes two series of multiscale features as input, one that is more sensitive to contrast information, and one that is able to select scale. The model combines them to classify each pixel in salient/nonsalient class, giving a binary segmentation of the image. The few parameters are learned with an EM-type algorithm.

This paper presents a formal model of an active recognition system that can be programmed by learning. At each time step the system decides between producing an action to generate new data and stopping to issue the name of the object observed. The actions can be directed either towards the external environment or towards the internal perceptual system of the agent. The decision strategy is based on a quantitative evaluation of the system learning experience. The problem studied is the recognition of chess pieces using a moving camera and a multiscale feature detector. The recognition is difficult because the objects are complex-neither polyhedral nor smooth-and rather similar between classes, especially in certain view configurations. The system uses the information obtained by observing internal state transitions when the camera is moved or when the feature detector scale is changed. A simulation of the agent and the environment is used for experimental measures of the model performances.