The video broadcasting of sports events leads daily to a mass of data. In this work, we are interested in the recognition of action phases (versus non-action, e.g. timeout) in videos of amateur sports competitions. The automation of this task can be related to the classification of actions from videos where the state of the art relies on the use of 3D deep neural approaches. Although providing good performances in the context of professional sports and offline analysis, deploying these approaches in an amateur/live processing setting can be problematic. In this study, we investigate the interest of an approach relying on intermediate local planar representations capturing both spatial and temporal information from the input videos. These representations are compatible with lightweight 2D neural models that are optimized for video classification. We evaluate the performances of the proposed approach combined with different visual backbones such as traditional convolutional neural networks (e.g., ResNet, MobileNet) as well as Transformer-based architectures and we compare our results with different baseline methods (2D and 3D models), on a dataset of 1000 annotated videos from amateur football competitions. Furthermore, we demonstrate the generalization capabilities of our method by evaluating it on a different sport (rugby) in a zero-shot setting. The results obtained highlight the interest of the proposed approach for sport competition videos, with acceptable performance and computation times allowing scaling in amateur/live processing settings.

Extracting good visual representations from image contents is essential for solving many computer vision problems (e.g. image retrieval , object detection, classification). In this context, state-of-the-art approaches are mainly based on learning a representation using a neural network optimized for a given task. The encoders optimized in this way can then be deployed as backbones for various downstream tasks. When the latter involves reasoning about spatial information from the image content (e.g. retrieve similar structured scenes or compare spatial configurations), this may be suboptimal since models like convolutional neural networks struggle to reason about the relative position of objects in images. Previous studies on building hand-crafted spatial representations , thanks to Relative Position Descriptors (RPD), showed they were powerful to discriminate spatial relations between crisp objects, but such spatial descriptors have rarely been integrated into deep neural networks . We propose in this article different strategies embedded in a common framework called SpIRL (SPatially-aware Image Representation Learning) to guide the optimization of encoders to make them learn more spatial information, under the supervision of an RPD and with the help of a novel dataset (44k images) that does not induce learning semantic information. By using these strategies, we aim to help encoders build more spatially-aware representations. Our experimental results showcase that encoders trained under the SpIRL framework can capture accurate information about the spatial configurations of objects in images on two selected downstream tasks and public datasets.

Visual Question Answering for Remote Sensing (RSVQA) is a task that aims at answering natural language questions about the content of a remote sensing image. The visual features extraction is therefore an essential step in a VQA pipeline. By incorporating attention mechanisms into this process, models gain the ability to focus selectively on salient regions of the image, prioritizing the most relevant visual information for a given question. In this work, we propose to embed an attention mechanism guided by segmentation into a RSVQA pipeline. We argue that segmentation plays a crucial role in guiding attention by providing a contextual understanding of the visual information, underlying specific objects or areas of interest. To evaluate this methodology, we provide a new VQA dataset that exploits very high-resolution RGB orthophotos annotated with 16 segmentation classes and question/answer pairs. Our study shows promising results of our new methodology, gaining almost 10% of overall accuracy compared to a classical method on the proposed dataset.

When dealing with high-dimensional multivariate time series classification problems, a well-known difficulty is the curse of dimensionality. In this article, we propose an original approach of transposition of multidimensional data into images to tackle the task of classification. We propose a lightweight hybrid model that take this transposed data as an input. This model contains convolutional layers as a feature extractor followed by a recurrent neural network. We apply our method to a large dataset consisting of individual patient medical records. We show that our approach allows us to significantly reduce the size of a network and increase its performance by opting for a transformation of the input data.

Remote sensing images carry a wealth of information that is not easily accessible to end-users as it requires strong technical skills and knowledge. Visual Question Answering (VQA), a task that aims at answering an open-ended question in natural language from an image, can provide an easier access to this information. Considering the geographical information contained in remote sensing images, questions often embed an important spatial aspect, for instance regarding the relative position of two objects. Our objective is to better model the spatial relations in the construction of a ground-truth database of image/question/answer triplets and to assess the capacity a VQA model has to answer these questions. In this article, we propose to use histograms of forces to model the directional spatial relations between geo-localized objects. This allows a finer modeling of ambiguous relationships between objects and to provide different levels of assessment of a relation (e.g. object A is slightly/strictly to the west of object B). Using this new dataset, we evaluate the performances of a classical VQA model and propose a curriculum learning strategy to better take into account the varying difficulty of questions embedding spatial relations. With this approach, we show an improvement in the performances of our model, highlighting the interest of embedding spatial relations in VQA for remote sensing applications.

Deep learning based techniques have been widely used for semantic segmentation. The underlying voluminous DNN models are trained on large datasets that have been annotated at the pixel level by humans. Such low-level annotation tasks are expensive to obtain for newly collected datasets. Alternatively, we propose ComViSe, a segmentation pipeline that requires only high-level annotations that remain relatively accessible (e.g., bounding boxes and labels of a detection, labels of a legend) to segment a given image. ComViSe embeds a segmentation framework, pre-trained on a semantically different dataset, to generate image region proposals. The pipeline relies then on several semantic, visual and geometric criteria to characterize each proposed region, and combines them to select the optimal segmentation mask, comparing diverse aggregation strategies from handcrafted formula to automatic ones, supervised or not. An experimental study conducted on the PASCAL VOC dataset shows that these effectively combined criteria are enough to select the mask proposals with the best IoU score in most cases, and that the aggregation can be done automatically.

Studying the spatial organization of objects in images is fundamental to increase both the understanding of a sensed scene and the explainability of the perceived similarity between images. This leads to the fundamental problem of handling spatial relations: given two objects depicted in an image, or two parts in an object, how to extract and describe efficiently their spatial configuration? Dedicated descriptors already exist for this task, like the efficient force histogram. In this article, we introduce the Force Banner, which extends it to two dimensions by using a panel of forces (attraction and repulsion), so as to benefit from more expressiveness and to model rich spatial information. This descriptor can be used as an intermediate representation of the image dedicated to the spatial configuration, and feed a classical 2D Convolutional Neural Network (CNN) to benefit from their powerful performances. As an illustration of this, we used it to solve a classification problem aiming to discriminate simple spatial relations, but with variable configuration complexities. Experimental results obtained on datasets of images with various shapes highlight the interest of this approach, in particular for complex spatial configurations.

This article focuses on the specific issue of drop caps image recognition in the context of cultural heritage preservation. Due to their heterogeneity and their weakly structured properties, these historical images represent challenging data. An important aspect in the recognition process of drop caps is their background styles, which can be considered as discriminative features to identify both the printer and the period. Most existing methods for style recognition are based on low-level features such as color or texture properties. In this article, we present a novel framework for the recognition of drop caps style based on features of higher levels. We propose to capture the spatial structure carried by these images using relative position descriptors modeling the enlacement between local cells of pixel layers obtained from a document segmentation step. Such descriptors are then exploited in an efficient bag-of-features learning procedure. Experimental results obtained on a dataset of historical drop caps images highlight the interest of this approach, and in particular the benefit of considering spatial information.

In this paper, we focus on analysis and diagnosis of breast masses inspired by expert concepts and rules. Accordingly, a Bag of Words is built based on the ontology of breast cancer diagnosis, accurately described in the Breast Imaging Reporting and Data System. To fill the gap between low level knowledge and expert concepts, a semantic annotation is developed using a machine learning tool. Then, breast masses are classified into benign or malignant according to expert rules implicitly modeled with a set of classifiers (KNN, ANN, SVM and Decision Tree). This semantic context of analysis offers a frame where we can include external factors and other meta-knowledge such as patient risk factors as well as exploiting more than one modality. Based on MRI and DECEDM modalities, our developed system leads a recognition rate of 99.7% with Decision Tree where an improvement of 24.7 % is obtained owing to semantic analysis.

The analysis of spatial relations between objects in digital images plays a crucial role in various application domains related to pattern recognition and computer vision. Classical models for the evaluation of such relations are usually sufficient for the handling of simple objects, but can lead to ambiguous results in more complex situations. In this article, we investigate the modeling of spatial configurations where the objects can be imbricated in each other. We formalize this notion with the term enlacement, from which we also derive the term interlacement, denoting a mutual enlacement of two objects. Our main contribution is the proposition of new relative position descriptors designed to capture the enlacement and interlacement between two-dimensional objects. These descriptors take the form of circular histograms allowing to characterize spatial configurations with directional granularity, and they highlight useful invariance properties for typical image understanding applications. We also show how these descriptors can be used to evaluate different complex spatial relations, such as the surrounding of objects. Experimental results obtained in the different application domains of medical imaging, document image analysis and remote sensing, confirm the genericity of this approach.

We introduce a novel bags-of-features framework based on relative position descriptors, modeling both spatial relations and shape information between the pairwise structural subparts of objects. First, we propose a hierarchical approach for the decomposition of complex objects into structural subparts, as well as their description using the concept of Force Histogram Decomposition (FHD). Then, an original learning methodology is presented, in order to produce discriminative hierarchical spatial features for object classification tasks. The cornerstone is to build an homogeneous vocabulary of shapes and spatial configurations occurring across the objects at different scales of decomposition. An advantage of this learning procedure is its compatibility with traditional bags-of-features frameworks, allowing for hybrid representations of both structural and local features. Classification results obtained on two datasets of images highlight the interest of this approach based on hierarchical spatial relations descriptors to recognize structured objects.

International audience

This paper focuses on breast masses analysis from two different modalities: Magnetic Resonance Imaging (MRI) and Dual-Energy Contrast Enhanced Digital Mammography (DECEDM). After the segmentation step, a set of texture and shape features are extracted from both MRI and DECEDM. Then textural and morphological information extracted from the two modalities are combined in order to improve breast cancer detection. Achieved results show that features combination extracted from two different breast images modalities can give a better characterization of breast cancer with a CCR of 96%.

In biomedical documents/publications, medical images tend to be complex by nature and often contain several regions that are annotated using arrows. In this context, an automated arrowhead detection is a critical precursor to region-of-interest (ROI) labeling and image content analysis. To detect arrowheads, in this paper, images are first binarized using fuzzy binarization technique to segment a set of candidates based on connected component (CC) principle. To select arrow candidates, we use convexity defect-based filtering, which is followed by template matching via dynamic time warping (DTW). The DTW similarity score confirms the presence of arrows in the image. Our test results on biomedical images from imageCLEF 2010 collection shows the interest of the technique, and can be compared with previously reported state-of-the-art results.

This paper deals with a complex symbol recognition process considering a large number of classes and only one training image per class. Furthermore, the response times of recognition system should be short and the interpretation of results must be easy. In this particular case, both statistical and structural methods are not the most suitable. A new composite descriptor and a similarity measure are proposed. Experimental results show the proposed method outperforms two descriptors widely used in symbol recognition with industrial data.

In this paper, we present a scalable arrow detection technique for biomedical images to support information retrieval systems under the purview of content-based image retrieval (CBIR) and text information retrieval (TIR). The idea primarily follows the criteria based on the geometric properties of the arrow, where we introduce signatures from key points associated with it. To handle this, images are first binarized via a fuzzy binarization tool and several regions of interest are labeled accordingly. Each region is used to generate signatures and then compared with the theoretical ones to check their similarity. Our validation over biomedical images shows the advantage of the technique over the most prominent state-of-the-art methods.

In this paper, we address a new scheme for symbol retrieval based on bag-of-relations (BoRs) which are computed between extracted visual primitives (e.g. circle and corner). Our features consist of pairwise spatial relations from all possible combinations of individual visual primitives. The key characteristic of the overall process is to use topological relation information indexed in BoRs and use this for recognition. As a consequence, directional relation matching takes place only with those candidates having similar topological configurations. A comprehensive study is made by using several different well-known datasets such as GREC, FRESH and SESYD, and includes a comparison with state-of-the-art descriptors. Experiments provide interesting results on symbol spotting and other user-friendly symbol retrieval applications.

In this paper, we address a new scheme for symbol retrieval based on relation bag-of-features (BOFs) which are computed between the extracted visual primitives. Our feature consists of pair wise spatial relations from all possible combinations of individual visual primitives. The key characteristic of the overall process is to use topological information to guide directional relations. Consequently, directional relation matching takes place only with those candidates having similar topological configurations. A comprehensive study is made by using two different datasets. Experimental tests provide interesting results by establishing user-friendly symbol retrieval application.

In this paper, we present a pattern recognition method that uses dynamic programming for the alignment of Radon features. The key characteristic of the method is to use dynamic time warping (DTW) to match corresponding pairs of the Radon features for all possible projections. Thanks to DTW, we avoid compressing the feature matrix into a single vector which would otherwise miss information. To reduce the possible number of matchings, we rely on a initial normalization based on the pattern orientation. A comprehensive study is made using major state-of-the-art shape descriptors over several public datasets of shapes such as graphical symbols (both printed and hand-drawn), handwritten characters and footwear prints. In all tests, the method proves its generic behavior by providing better recognition performance. Overall, we validate that our method is robust to deformed shape due to distortion, degradation and occlusion.

This paper presents a study on merging confidence measures using fuzzy logic. Instead of the previous approaches using the notion of probability, we propose to observe the uncertainty of the recognition hypotheses and the notion of possibility thanks to fuzzy reasoning. Four different confidence measures are developed, coming from different parts of a speech recognizer. Various merging methods are studied to improve the performance of the confidence measures. The methods are evaluated in terms of Confidence Error Rate (CER) and in terms of their Detection Error Tradeoff (DET) curves on a French broadcast news corpus. They are compared to some fuzzy logic aggregation techniques among which the technique based on the Choquet Integral yields to a significant improvement in terms of CER.

An iterative method to select suitable features for pattern recognition context has been proposed (FRIFS). It combines a global feature selection method based on the Choquet integral and a fuzzy linguistic rule classifier. In this paper, enhancements of this method are presented. An automatic step has been added to make it adaptive to process numerous features. The experimental study, made in a wood defect recognition context, is based on several classifier result analysis. They show the relevancy of the remaining set of selected features. The recognition rates are also considered for each class separately, showing the good behavior of the proposed method.

A new method to extract dashed lines in technical document is proposed in this paper by combining force histogram and discrete lines. The aim is to study the spatial location of couples of connected components using force histogram and to refine the recognition by considering surrounding discrete lines. This new model is fast and it allows a good extraction of occulted patterns in presence of noise. Efficient common methods require several thresholds to process with technical documents. The proposed method requires only few thresholds which can be automatically set from data.

In this paper, we address the use of unified spatial relations for symbol description. We present a topologically guided directional relation signature. It references a unique point set instead of one entity in a pair, thus avoiding problems related to erroneous choices of reference entities and preserves symmetry. We experimentally validate our method on showing its ability to serve in a symbol retrieval application, based only on a spatial relational descriptor that represents the links between the decomposed structural patterns called "vocabulary" in a spatial relational graph.

An original binarization method based on connected operators is proposed in this paper. Connected operators enable to filter and/or segment an image by preserving its contours.The proposed binarization method enables to extract relevant document objects by means of the component-tree structure. This method was compared to other binarization methods and showed good behavior in various contexts.

We present an approach to automatically extract a pertinent subset of soft output classifiers, and to aggregate them into a global decision rule using the Choquet integral. This approach relies on two key points. The first is a learning algorithm that uses a measure of the confusion between the categories to be recognized. The second is a selection scheme that discards weak or redundant decision rules, keeping only the most relevant subset. An experimental study, based on real world data, is then described. It analyzes the improvements achieved by these points first when used independently, then when combined together.

In this paper, an iterative method to select suitable features in an industrial pattern recognition context is proposed. It combines a global method of feature selection and a fuzzy linguistic rule classifier. It is applied to an industrial fabric textile context. The aim of the global vision system is to identify textile fabric defects. From the related industrial process, the training data sets are small, and some are incomplete. Moreover, the recognition step must be compatible with the time constant of the system, which generally imposes low complexity for the system. The choice of the most relevant features and the reduction of their number are important to respect these constraints. The feature selection method is based on the analysis of indexes extracted on the lattice defined from training in relation with the Choquet integral. This selection step is embedded in an iterative algorithm to discard weaker features in order to decrease the number of rules while keeping good recognition rates. The recognition step is done with a fuzzy reasoning classifier that is well adapted for this application case. The proposed method is quite efficient with small learning data sets because of the generalization capacity of both the feature selection and recognition steps. The experimental study shows the wanted behavior of this approach: the feature number decreases, whereas the recognition rate increases. Thus, the total number of generated fuzzy rules is reduced.

An iterative feature selection method based on feature typicality and interactivity analysis is presented in this paper. The aim is to enhance model interpretability by selecting the best significant features among a list extracted from images. The inference mechanism uses a fuzzy linguistic rule-based system. This method is applied here to a wood defect classification problem. Nowadays, feature selection is expertise-driven and most of the time, expert uses features by habits which not always represent the best ones to use. The proposed approach aims to replace expert selection by automatically choosing a suitable set of features to the recognition problem.

We present an approach allowing to automatically extract a suitable set of soft output classifiers and to aggregate them to provide a global decision using the Choquet integral. This approach relies on two key points. A learning algorithm based on a 2-class model is performed to define a new set of decisions rules assuming to be experts dedicated to recognize one class from another one. All the associated capacities are aggregated again at a high level to recognize symbols. The second is a selection scheme that discards weak or redundant decision rules, keeping only the most relevant subset. An experimental study, based on real world data, is then described. It analyzes the improvements achieve by these points first when used independently, then when combined together.

This paper presents a novel approach to combine shape descriptors. Each approach is applied on several clusters of objects. For each cluster and for any descriptor a map is associated directly from the confusion matrix. Such a method allows to determine automatically the better weight associated to the descriptor for the object under consideration. At last, we show that the additive combination of such measures allows to improve the classification.

A new way of detecting arrows in line drawings is proposed in this paper. We provide a set of criteria which are aggregated using the Choquet integral. These criteria are defined from the geometric properties of an arrow. Experimental results on two kinds of line-drawing documents show the interest of our approach.

A new way to detect arrows in line drawings is proposed in this paper. Our approach is based on the definition of the structure of such a symbol. Signatures of angular areas are computed and axiomatic properties and geometric characteristics are checked using the Choquet integral. Finally an experimental application on line-drawing documents shows the interest of our approach.

We propose a method for indexing technical drawings. Our features are based on the notion of F-signature, which is a particular histogram of forces. The force histogram has low time complexity and describes a signature which is invariant to scaling, translation, symmetry and rotation. This article presents a new application of such signatures in the field of document analysis, and tests different characteristics of the F-signatures, like their sensitivity to the shape of the objects and to noise. Finally, experimental results show the effectiveness of such an approach. A brief overview of pattern recognition approaches dedicated to technical document indexing is given. The notions of histogram of forces and of its properties are presented.

Fuzzy set methods have been used to model and manage uncertainty in various aspects of image processing, pattern recognition, and computer vision. High-level computer vision applications hold a great potential for fuzzy set theory because of its links to natural language. Linguistic scene description, a language-based interpretation of regions and their relationships, is one such application that is starting to bear the fruits of fuzzy set theoretic involvement. In this paper, we are expanding on two earlier endeavors. We introduce new families of fuzzy directional relations that rely on the computation of histograms of forces. These families preserve important relative position properties. They provide inputs to a fuzzy rule base that produces logical linguistic descriptions along with assessments as to the validity of the descriptions. Each linguistic output uses hedges from a dictionary of about 30 adverbs and other terms that can be tailored to individual users. Excellent results from several synthetic and real image examples show the applicability of this approach.

In a previous work (1999), we defined a new approach to evaluating directional relations between objects. The approach is based on the computation of a histogram of forces. We show in the present paper that the notion of the histogram of forces can also be useful in pattern recognition. The aim of this this study is to classify orbits and sinuses. Using force histograms instead of geometrical criteria (like compactness or degree of ellipticity) allows more accurate classification to be produced. Moreover, the proposed approach requires low completing time.

The fuzzy qualitative evaluation of directional spatial relationships (such as "to the right of", "to the south of...") between areal objects often relies on the computation of a histogram of angles, which is considered to provide a good representation of the relative position of an object with regard to another. In this paper, the notion of the histogram of forces is introduced. It generalizes and may supersede the histogram of angles. The objects (2D entities) are handled as longitudinal sections (1D entities), not as points (OD entities). It is thus possible to fully benefit from the power of integral calculus and, so, ensure rapid processing of raster data, as well as of vector data, explicitly considering both angular and metric information.

In this paper, a new method of pattern recognition based on images splitting into a set of trees composed of fuzzy regions is presented. First, either a gradient inverse function is applied on the raster image to define the fuzzy regions supports, or we manage with the basic grey level image if regions are easily topologically separable. Then, topologic features are computed on these sets. Therefore, a tree description of the image, which consists of fuzzy regions with associated topological features, is obtained. A set of sample trees is achieved from the application of the fuzzy segmentation algorithm on characteristic objects (small images). Then a tree isomorphism is defined to recognize a particular object in an image. At last, a new tree compression method is introduced in order to decrease the complexity when we have to manage with a large set of samples.