The evaluation of Retrieval-Augmented Generation (RAG) systems typically examines retrieval quality and generation parameters like temperature in isolation, overlooking their interaction. This work presents a systematic investigation of how text perturbations (simulating noisy retrieval) interact with temperature settings across multiple LLM runs. We propose a comprehensive RAG Perturbation-Temperature Analysis Framework that subjects retrieved documents to three distinct perturbation types across varying temperature settings. Through extensive experiments on HotpotQA with both open-source and proprietary LLMs, we demonstrate that performance degradation follows distinct patterns: high-temperature settings consistently amplify vulnerability to perturbations, while certain perturbation types exhibit non-linear sensitivity across the temperature range. Our work yields three key contributions: (1) a diagnostic benchmark for assessing RAG robustness, (2) an analytical framework for quantifying perturbation-temperature interactions, and (3) practical guidelines for model selection and parameter tuning under noisy retrieval conditions.

While the Segment Anything Model (SAM) has demonstrated remarkable capabilities in general image segmentation, its direct application to medical images faces significant challenges, leading to domain-specific adaptations like MedSAM. In current medical literature, SAM-like models are often evaluated using “simulated” bounding box prompts derived from ground truth annotations, potentially overestimating their real-world performance. This paper addresses this underexplored problem by systematically evaluating the influence of bounding box quality on melanoma segmentation performance using the ISIC 2018 dataset (RGB skin image corpus). Experiments were conducted on both the foundational SAM and MedSAM, utilizing a structured protocol that includes ideal bounding boxes, baseline heuristic prompts (full image, most probable location, centered point for SAM), and controlled perturbations. These perturbations comprise five levels of expanding, shrinking, and shifting the ideal bounding box (random disturbance between 5% and 25%). Our results demonstrate a significant dependence of segmentation performance on bounding box quality for both models. Depending on the type of bounding box perturbation, the Dice score decreases from 84% under ideal conditions to 37% with inaccurate bounding boxes, corresponding to a reduction of more than 50%. Notably, shrinking the bounding box has a more detrimental effect than expanding it, with accuracy dropping substantially (e.g., MedSAM's Dice score decreases from 91.5 to 60.9 under a 25% shrinkage). Moreover, medical pre-training significantly improves robustness, as MedSAM outperforms SAM by approximately 7 Dice points under ideal conditions. These findings highlight that although SAM-like models offer promising capabilities, their reliance on high-quality bounding box prompts currently limits their applicability for fully unsupervised segmentation in medical image processing pipelines. This underscores the need for either human intervention or more advanced automated prompting strategies in clinical applications.

Segmentation of medical images using learning-based systems remains a challenge in medical computer vision, as training a segmentation model requires exhaustively annotated medical images by experts, which are difficult and expensive to obtain. In the context of melanoma segmentation, we explored active learning methods to combine human annotation for the most uncertain pixels with model predictions for the others. With only around$30 \%$of the images annotated, we achieved performance similar to that obtained with a fully annotated dataset. We also demonstrated that, after a few iterations, experts can focus on annotating only the most uncertain areas of the images, relying on the model for the rest. These approaches pave the way for accelerating the annotation of unlabeled medical datasets and optimizing the use of medical expertise in deep learning projects.

This paper focuses on the causal relation between the detection scores of concept (or tag) classifiers and the ranking decisions based on these scores, paving the way for these tags to be used in the visual explanations. We first define a measure for quantifying a causality on a set of tags, typically those involved in visual explanations. We use this measure for evaluating the actual causality in the explanations generated using a recent interpretable video retrieval system (Dong et al. [4]), which we find to be quite low. We then propose and evaluate improvements for significantly increasing this causality without sacrificing the retrieval accuracy of the system.

LongEval-Retrieval is a Web document retrieval benchmark that focuses on continuous retrieval evaluation. This test collection is intended to be used to study the temporal persistence of Information Retrieval systems and will be used as the test collection in the Longitudinal Evaluation of Model Performance Track (LongEval) at CLEF 2023. This benchmark simulates an evolving information system environment - such as the one a Web search engine operates in - where the document collection, the query distribution, and relevance all move continuously, while following the Cranfield paradigm for offline evaluation. To do that, we introduce the concept of a dynamic test collection that is composed of successive sub-collections each representing the state of an information system at a given time step. In LongEval-Retrieval, each sub-collection contains a set of queries, documents, and soft relevance assessments built from click models. The data comes from Qwant, a privacy-preserving Web search engine that primarily focuses on the French market. LongEval-Retrieval also provides a 'mirror' collection: it is initially constructed in the French language to benefit from the majority of Qwant's traffic, before being translated to English. This paper presents the creation process of LongEval-Retrieval and provides baseline runs and analysis.

This paper introduces a novel visualization tool that facilitates the exploratory analysis of continuous evaluation for information retrieval systems. We base our analysis on score standardization and meta-analysis techniques applied to Information Retrieval evaluation. We present three functionalities: evaluation overview, delta evaluation, and meta-analysis applied to three perspectives: evaluation rounds, queries, and systems. To illustrate the use of the tool, we provide an example using the TREC-COVID test collection.

This paper focuses on studying the complementarity between the spaces from hybrid cross-modal state-of-the-art systems for video retrieval like [5]. We aim at investigating if these spaces really convey different features, or if they are representing the same things. We use PCA (Principal Component Analysis) to study the optimal dimensions, CCA (Canonical Correlation Analysis) to assess the similarity of the spaces, and check if such approach is in fact similar to ensemble learning. We achieve experiments on the MST-VTT corpus, and show that in fact these two spaces are indeed very similar, paving the way for new models that could enforce more dissimilar spaces.

International audience

Document indexing is a key component for efficient information retrieval (IR). After preprocessing steps such as stemming and stop-word removal, document indexes usually store term-frequencies (tf). Along with tf (that only reflects the importance of a term in a document), traditional IR models use term discrimination values (TDVs) such as inverse document frequency (idf) to favor discriminative terms during retrieval. In this work, we propose to learn TDVs for document indexing with shallow neural networks that approximate traditional IR ranking functions such as TF-IDF and BM25. Our proposal outperforms, both in terms of nDCG and recall, traditional approaches, even with few positively labelled query-document pairs as learning data. Our learned TDVs, when used to filter out terms of the vocabulary that have zero discrimination value, allow to both significantly lower the memory footprint of the inverted index and speed up the retrieval process (BM25 is up to 3~times faster), without degrading retrieval quality.

The performance of deep Convolutional Neural Networks (CNN) has been reaching or even exceeding the human level on large number of tasks. Some examples are image classification, Mastering Go game, speech understanding etc. However, their lack of decomposability into intuitive and understandable components make them hard to interpret, i.e. no information is provided about what makes them arrive at their prediction. We propose a technique to interpret CNN classification task and justify the classification result with visual explanation and visual search. The model consists of two sub networks: a deep recurrent neural network for generating textual justification and a deep convolutional network for image analysis. This multimodal approach generates the textual justification about the classification decision. To verify the textual justification, we use the visual search to extract the similar content from the training set. We evaluate our strategy on a novel CUB dataset with the ground-truth attributes. We make use of these attributes to further strengthen the justification by providing the attributes of images.

Single visual concept detection in videos is a hard task, especially for infrequent concepts or for those difficult to model. This question becomes even more difficult in the case of concept pairs. Two main directions may tackle this problem: 1) combine the predictions of their corresponding detectors in a way which is similar to usual information retrieval, or 2) build supervised learners for these pairs of concepts by generating annotations based on the occurrences of the two individual concepts. Each of these approaches have advantages and drawbacks. We evaluated them in the context of the concept pair detection subtask of the TRECVid 2013 semantic indexing (SIN) task and found that information retrieval-like fusions of concept detection scores outperforms the learning approaches. The described methods outperform the best official result of the evaluation campaign cited previously, by 9% in terms of relative improvement on MAP.

Regions Of Interest (ROI) are successfully used in automatic image annotation through Bag of Visual Words (BoVW) models. However, the "flat" BoVW models ignore potentially relevant topological relationships between the ROI. While several works tried to take into account such relations for Content Based Image Retrieval (CBIR), few works evaluated clearly their effects in the context of automatic image annotation. In this article, we propose to explicit the relationships between ROI in automatic annotation methods in order to evaluate their contributions on the results quality. This is done through a model, called "phrasing model", that generates groups of ROI called Visual Phrases according to relationships constraints. Based on our phrasing model, we define two kinds of Visual Phrases and we show that taking into account topological ROI relationships can significantly overcome the results of the classical BoVW model.

In this paper, a language model adapted to graph-based representation of image content is proposed and assessed. The full indexing and retrieval processes are evaluated on two different image corpora. We show that using the spatial relationships with graph model has a positive impact on the results of standard Language Model (LM) and outperforms the baseline built upon the current state-of-the-art Support Vector Machine (SVM) classification method.

This article addresses the problem of processing the annotations of preexisting video productions to enable reuse and repurposing of metadata. We introduce the concept of automatic content-based editing of preexisting semantic home video metadata. We propose a formal representation and implementation techniques for reusing and repurposing semantic video metadata in concordance with the actual video editing operations. A novel representation for metadata editing is proposed and an implementation framework for editing the metadata in accordance with the video editing operations is demonstrated. Conflict resolution and regularization operations are defined and implemented in the context of the video metadata editing operations.

The paper presents a contribution to image indexing consisting in a weighting model for visible objects -- or image objects -- in home photographs. To improve its effectiveness this weighting model has been designed according to human perception criteria about what is estimated as important in photographs. Four basic hypotheses related to human perception are presented, and their validity is estimated as compared to actual observations from a user study. Finally a formal definition of this weighting model is presented and its consistence with the user study is evaluated.

Bridging the 'signal-symbol gap' existing between multimedia signals generated through audio, video or other multimedia streams and the high level symbols (metadata) which describe them is presently one of the most vital areas of multimedia research. The paper attempts to bridge this significant gap by proposing a novel automatic mechanism for XML based video metadata editing, in tandem with video editing operations. An implementation framework for editing metadata in accordance with the video editing operations is demonstrated. Conflicting resolution and regularization operations are defined and implemented with respect to video metadata editing operations.

Faces usually are the most interesting objects in certain categories of video, like home videos and news clips. A novel sensor fusion based face tracking system is presented that tracks faces in compressed video, and aids automatic video indexing. Tracking is done by fusing the measurements from three independent sensors - motion and colour based trackers (Achanta, R. et al., IEEE Int. Conf. on Multimedia and Expo, 2002) and a face detector (Wang, J. et al., Proc. Int. Workshop on Advanced Image Technology, 2002) using a novel hierarchical framework based on Kalman filter state vector fusion. The tracking results show that the fused results are better than those of any individual sensors or their mean.

With rapid advances in sensor, storage, processor, and communication technologies, consumers can now afford to create, store, process, and share large digital photo collections. With more and more digital photos accumulated, consumers need effective and efficient tools to organize and access photos in a semantically meaningful way without too much manual annotation effort. From user studies, we confirm that users prefer to organize and access photos along semantic axes such as event, people, time, and place. In this paper, we propose a computational learning framework to construct event models from sample photos with event labels given by a user and to compute relevance measures of unlabeled photos to the event models. We demonstrate event-based retrieval on 2400 genuine home photos using our proposed approach.

In this paper, we present a semantic summarization algorithm that interfaces with the metadata and that works in compressed domain, in particular MPEG-1 and MPEG-2 videos. In enabling a summarization algorithm through high-level semantic content, we try to address two major problems. First, we present the facility provided in the DVA system that allows the semi-automatic creation of this metadata. Second, we address the main point of this system which is the utilization of this metadata to filter out frames, creating an abstract of a video summary quality survey indicates that the proposed method performs satisfactorily.

Photograph retrieval systems face the difficulty to deal with the different ways to apprehend the content of images. We consider and demonstrate here the use of multiple index representations of photographs to achieve effective retrieval. The use of multiple indexes allows integration of the complementary strengths of different indexing and retrieval models. The proposed representation supports multiple labels for regions and attributes, and handles inferences and relationships. We define links between indexing levels and the related query modes. The experiment conducted on 2400 home photographs shows the behavior of the multiple indexing levels during retrieval.

Object tracking is of utmost importance for automatic indexing of video content. This work presents an object tracker that operates directly on MPEG compressed data. Motion vectors and discrete cosine transform (DCT) coefficients directly available from the compressed video stream are exploited for the purpose of tracking. Tracking proceeds in two steps: motion vector based tracking in P and B frames within the groups of pictures (GOPs), and object identification in I frames. Colour, which is one of the strongest cues for tracking is used for the identification step. Such a system offers speed, simplicity and robustness against occlusion and camera motion, with good intra-shot tracking for shots in excess of 500 frames, as shown in the experimental results.

This demonstration presents a novel application (called SmartAlbum) for photo indexing and retrieval that unifies two different image indexing approaches. The system uses two modalities to extract information about a digital photograph; i.e. content-based and speech annotation for image description. The result is a powerful image retrieval tool that has capabilities beyond what current single-mode retrieval systems can offer. We show on a corpus of 1200 images the interest of our approach.

This paper presents a novel image-based indexing scheme called object probes. Object probes make use of both physical features and a manually formulated decision tree to classify objects in the scene. A conceptual graph is used to represent the association of the labels found in each labeled image. We develop a query formalism that translates the query elements into a graph structure. The query process can be seen as matching the query graph with those sub-graphs found in the database of images.

No abstract available.

Article Interactive information retrieval systems: from user centered interface design to software design Share on Authors: P. Mulhem Laboratoire CLIPS-IMAG, Grenoble, France Laboratoire CLIPS-IMAG, Grenoble, FranceView Profile , L. Nigay Laboratoire CLIPS-IMAG, Grenoble, France Laboratoire CLIPS-IMAG, Grenoble, FranceView Profile Authors Info & Claims SIGIR '96: Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrievalAugust 1996 Pages 326–334https://doi.org/10.1145/243199.243280Online:18 August 1996Publication History 7citation1,125DownloadsMetricsTotal Citations7Total Downloads1,125Last 12 Months6Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

This paper presents outlines to express the matchmg between complex objects, using works done in the Information Retrieval field.The basic idea is to help an application programmer to describe theorcticatly the matching using modal logic, and then to express the operational matching.We describe here these two parts using a medical information retrieval example.We are implementing our work on the 02 system [1].