Contrastive Language-Image Pre-training (CLIP) has demonstrated strong potential in learning transferable visual models / representations by aligning paired images and textual descriptions. Nevertheless, the quality of training data remains a significant bottleneck. In many real-world scenarios, image-text pairs are often noisy or accompanied by captions that are either too short or too generic to convey key visual attributes. For example, in medical imaging, most available data come from illustrative figures in public literature instead of detailed clinical reports, resulting in captions that lack the precision and context provided by expert annotations. Recent efforts to improve caption quality using large language models have largely focused on natural images and overlook the integration of domain-specific knowledge. In this study, we propose a retrieval-augmented generation framework guided by expert semantic knowledge to enrich image captions in the medical context. We further introduce a multi-text training strategy that effectively incorporates these enriched descriptions into CLIP training. Our approach, demonstrated in the medical domain as a proof of concept, achieves state-of-the-art performances on multiple downstream tasks, highlighting its broader potential for vision-language pretraining in specialized domains. Our code is available at: https://github.com/Wxy-24/RaceCLIP.

By aligning paired image and caption embeddings as input, contrastive vision-language representation learning has witnessed significant advances as illustrated by CLIP, allowing visual encoders to learn from textual supervision and vice versa. Benefiting from millions of image-caption pairs collected from the Internet, CLIP-like models show competitive performances against fully supervised baselines. However, the learned visual representations are still undermined due to the binary constraint, as most contrastive learning frameworks follow strict one-to-one corre-spondence for the input pairs of data and optimize the models using the InfoNCE loss function. The embeddings of the paired image-text are aligned while the unpaired image-text are pushed away from each other. In fact, there are natu-rally many “false negatives” among these negative pairs since unpaired data can also have a high similarity. In this work, we aim to overcome the impact offalse negatives in vision-language representation learning by introducing soft targets for estimating the similarity between unpaired images and texts using external semantic knowledge structured in the form of graphs. The interest of such a method is demonstrated in the application context of medical imaging.

Visual representation learning plays a key role in solving medical computer vision tasks. Recent advances in the literature often rely on vision-language models aiming to learn the representation of medical images from the supervision of paired captions in a label-free manner. The training of such models is however very data/time intensive and the alignment strategies involved in the contrastive loss functions may not capture the full richness of information carried by inter-data relationships. We assume here that considering expert knowledge from the medical domain can provide solutions to these problems during model optimization. To this end, we propose a novel knowledge-augmented vision-language contrastive representation learning framework consisting of the following steps: (1) Modeling the hierarchical relationships between various medical concepts using expert knowledge and medical images in a dataset through a knowledge graph, followed by translating each node into a knowledge embedding; And (2) integrating knowledge embeddings into a vision-language contrastive learning framework, either by introducing an additional alignment loss between visual and knowledge embeddings or by relaxing binary constraints of vision-language alignment using knowledge embeddings. Our results demonstrate that the proposed solution achieves competitive performances against state-of-the-art approaches for downstream tasks while requiring significantly less training data. Our code is available at https://github.com/Wxy-24/KL-CVR .

Content-based image retrieval (CBIR), which consists of ranking a set of images with respect to a query image based on visual similarity, can assist diagnostic radiologists in assessing medical images, by identifying similar digital images in large image databases. Despite the many recent advances and innovations in CBIR for general images, their adoption in radiology has been slow and limited. In the current paper we attempt to close the gap between the two domains and wisely adapt modern CBIR techniques to radiology images: by extending the latest representation learning techniques in a way that can overcome the unique challenges and at the same time take advantage of the specific opportunities that are present in radiology we were able to come up with novel and effective medical image retrieval methods. Our method achieves the highest CUI@5 scores (18.48, 15.95) on two widely used datasets (ROCO and MEDICAT respectively), showcasing the superiority of the proposed method in comparison with state-of-the-art relevant alternatives. • Metadata about image semantics is beneficial for representation learning. • External medical knowledge can be integrated for better semantics understanding. • Severity of mistakes can to be taken into account to improve image retrieval. • Differentiable average precision-based loss is suitable for representation learning.

Radiologists are now confronted with the difficulty of interpreting cross-sectional studies composed of thousands of images. In hospitals, all acquired imaging data are stored in a picture archiving and communication system (PACS). To take advantage of these masses of previously interpreted images, with the ultimate goal to facilitate the diagnosis of new cases, a promising approach would be to integrate a Content-Based Image Retrieval (CBIR) system into PACS. CBIR system performances are inherently limited by the features considered to represent the images. The current state of the art for the extraction of visual features relies on deep learning which requires a sufficient amount of annotated data to learn a generalizable model, such annotated data being rare and difficult to use in medical imaging. At the same time, PACS contain additional information such as radiological reports which supplement visual information carried by the images. We study here how such semantic information, hidden in these reports, can be used to supervise the learning of neuronal models to build a better visual representation of images. In this context our contribution is threefold. We first adapted a contrastive learning approach, which is usually used to learn representation from pairs of positive images in an unsupervised manner, to deal with in-domain medical data. Second, to train such a model to be robust and generalizable with a sufficient amount of data, we propose to re-employ the "dormant" medical imaging literature. Finally, the visual features and the deep models learned in this way, can be considered in CBIR systems as coarse-grained information which can then be fine-tuned in PACS, with more specific images depending on the applications. The obtained experimental result with state of the art contrastive learning methods highlight the interest of this approach.

Document layout extraction is a difficult step in the image interpretation process due to the high complexity of documents. The main challenge relies on the huge gap between both the physical and the logical structures of document images. In order to loose as few as possible information, most existing methods are working at pixel level. In this paper, we present a new framework for complex layout extraction based on features of high levels obtained from a document straight line based segmentation. We propose to capture the straight line segments thanks to a new transform integrating the local spatial organization of the segments contained in the document content. Such transform can be applied either on the foreground (related to the document content) or the background pixels, in order to take advantage of the duality of information present in both document parts. Experimental results obtained on the PRImA Layout Analysis dataset illustrate the robustness of our framework for the extraction of specific components of the document including text areas, images and separators.

This paper presents the results of the ICDAR2017 Competition on the Classification of Medieval Handwritings in Latin Script (CLaMM), jointly organized by Computer Scientists and Humanists (paleographers). This work follows a competition at ICFHR2016 and aims at providing a rich annotated database of European medieval manuscripts to the community on Handwriting Analysis and Recognition. We proposed four independent classification tasks which attracted 10 registered teams, with 6 submitted classifiers from 4 participants. Those classifiers are trained on a set of 3540 images with their ground truths. In task 1 (Script classification) and task 3 (Date classification), the classifiers have been evaluated by a test set of 2000 greyscale, tiff, 300 dpi images. In task 2 (Script classification) and task 4 (Date classification), the test set consists of 1000 images in different formats, resolutions and color representation. The best scores are respectively 85.2% for task 1, 76.5% for task 2, 59% for task 3, and 49.9% for task 4. An analysis based on the matrix of confusion of each classifier is also given.

In this paper, we focus on analysis and diagnosis of breast masses inspired by expert concepts and rules. Accordingly, a Bag of Words is built based on the ontology of breast cancer diagnosis, accurately described in the Breast Imaging Reporting and Data System. To fill the gap between low level knowledge and expert concepts, a semantic annotation is developed using a machine learning tool. Then, breast masses are classified into benign or malignant according to expert rules implicitly modeled with a set of classifiers (KNN, ANN, SVM and Decision Tree). This semantic context of analysis offers a frame where we can include external factors and other meta-knowledge such as patient risk factors as well as exploiting more than one modality. Based on MRI and DECEDM modalities, our developed system leads a recognition rate of 99.7% with Decision Tree where an improvement of 24.7 % is obtained owing to semantic analysis.

In this paper, we propose an OCR accuracy prediction method based on a local blur estimation since blur is one of the important factors that mostly damage OCR accuracy. First, we apply the blur estimation on synthetic blurred images by using Gaussian and motion blur in order to investigate the relation between blur effect and character size regarding OCR accuracy. This relation is considered as a blur-character size feature to define a classifier. Finally, the classifier can separate characters of a given document into three classes: readable, intermediate, and non-readable classes. Therefore, the quality score of the document is inferred from the three classes. The proposed method is evaluated on a published database and on an industrial one. The correlation with OCR accuracy is also given to compare with the state-of-the-art methods.

This paper presents the results of the ICFHR2016 Competition on the Classification of Medieval Handwritings in Latin Script (CLaMM), jointly organized by Computer Scientists and Humanists (paleographers). This work aims at providing a rich database of European medieval manuscripts to the community on Handwriting Analysis and Recognition. At this competition, we proposed two independent classification tasks which attracted five participants with seven submitted classifiers. Those classifiers are trained on a set of 2000 images with their ground truths. In the first task of script crisp classification, the classifiers have been evaluated on a test set of 1000 single-type manuscripts. In the second task of "Fuzzy Classification", the classifiers have been carried out on a set of 2000 multi-script-type manuscripts. The results of the participants provide the first baseline evaluation up to the accuracy score of 83.9% for the task 1 and to the fuzzy weighted score of 2.96/4 for the task 2. An analysis based on the intra-class distance and matrix of confusion of each classifier is also given.

Image quality is a hot topic since most of image analysis and recognition systems are sensitive to degradation. In this paper, we are interested in the quality of document images to recover text content with as few errors as possible. One of the defects that degrades the Optical Character Recognition rate (OCR) is blur. Therefore, we propose to detect blur and to attenuate its effect on an OCR result. As blur is non-uniform on the document area, we propose a local approach. No prior model is chosen for blur that may be of various natures. Thanks to a local clustering based on a novel blur feature, we build a debluring adapted to the heterogeneous blur. As a result, the blurred image is locally corrected according to the blur type. This method focuses on the ease of character segmentation, then makes OCR more efficient. The experiments are carried out on a public database DIQA together with the use of an OCR. The obtained results show that the proposed method gives an overall improvement of 11% in OCR rate.

International audience

Feature selection happens to be an important step in any classification process. Its aim is to reduce the number of features and at the same time to try to maintain or even improve the performance of the used classifier. Variability of handwriting makes features more or less efficient and gives a good support for evaluation of selection method. The selection methods described in the literature present some limitations at different levels. Some are too complex or too dependent on the classifier used for evaluation. Others overlook interactions between features. In this paper, we propose a fast selection method based on a genetic algorithm. Each feature is closely associated with a single feature classifier. The weak classifiers we consider have several degrees of freedom and are optimized on the training dataset. The classifier subsets are evaluated by a fitness function based on a combination of single feature classifiers. Results on the MNIST handwritten digits database show how robust our approach is and how efficient the method is.

This paper aims to develop a methodology to register pairs of temporal mammograms. Control points based on anatomical features are detected in an automated way. Thereby, image semantic is used to extract landmarks based on these control points. A referential is generated from these control points based on this referential the studied images are realigned using different levels of observation leading to both rigid and pseudo non-rigid transforms according to expert mammogram reading.

This paper presents a generic features selection method and its applications on some document analysis problems.The method is based on a genetic algorithm (GA), whose fitness function is defined by combining Adaboot classifiers associated with each feature. Our method is not linked to a classifier achieving the final recognition task; we have used a combination of weak classifiers to evaluate a subset of features. So we select features that can further be used in the most appropriate classifiers.This method has been tested on three applications: dropcaps classification, handwritten digits recognition and text detection. The results show the efficiency and robustness of the proposed approach.

This paper presents a fast method using simple genetic algorithms (GAs) for features selection. Unlike traditional approaches using GAs, we have used the combination of Adaboost classifiers to evaluate an individual of the population. So, the fitness function we have used is defined by the error rate of this combination. This approach has been implemented and tested on the MNIST database and the results confirm the effectiveness and the robustness of the proposed approach.

This paper presents a method of overlapping/aggregating nuclei cells segmentation. This method is based on the watershed segmentation algorithm, but the specificity of this work is to introduce some prior information about the usual shape of normal/pathological nuclei cells. Such prior information will help to optimize the right set of markers, from which the flooding will be done. This approach has been implemented and tested, and the results evaluated by cell biology experts show the efficiency of the proposed approach.

One of the major goals of computer vision is the research and the development of flexible methods for shape description. A large group of shape description techniques is given by heuristic approaches, which yield acceptable results in the description of simple shapes and regions. In this case, objects are represented by a planar graph with nodes symbolizing subregions from region decomposition, and region shape is then described by the graph properties. In the paper, the angular bisector network (ABN), a descriptor of polygonal shape, is used to automatically detect intersections between neurites of cell structures. Some properties of the ABN, such as linear algebraic complexity, easy extraction of characteristic points, etc., are very useful and experimental results are promising.