Biometric Verification Systems (BVS) often suffer from misclassification errors, which are frequently concentrated around a small subset of users for whom the system performs poorly. The Biometric Menagerie was introduced to categorize such users based on their biometric behavior. However, its main representation, the ZooPlot, fails to accurately distinguish all categories defined in the original Menagerie, particularly "Lambs" (easily impersonated) and "Wolves" (frequently impersonate others).This paper proposes two new visualizations for evaluating BVS: Inter ZooPlot and the Biometric Confusion Matrix (BCM). A user study was conducted to assess the effectiveness of different visualizations. Inter ZooPlot and BCM achieved average accuracies of 90.0% and 89.4%, respectively, in distinguishing the four original categories defined in the Biometric Menagerie, outperforming the baseline ZooPlot at 73.9%. Furthermore, we show that BCM can reveal sources of user-specific errors and highlight system imbalances, making it a promising post-hoc explainability method for biometric system analysis. All additional materials are available at: https://github.com/Boyu1998/BCM.

Due to their great performance in many challenges, Deep Learning (DL) techniques keep gaining popularity in many fields. They have been adapted to process graph data structures to solve various complicated tasks such as graph classification and edge prediction. Eventually, they reached the Graph Drawing (GD) task. This article is an extended version of the previously published(DNN)2and presents a framework to leverage DL techniques for graph drawing (DL4GD). We demonstrate how it is possible to train a Deep Learning model to extract features from a graph and project them into a graph layout. The method proposes to leverage efficient Convolutional Neural Networks, adapting them to graphs using Graph Convolutions. The graph layout projection is learned by optimizing a cost function that does not require any ground truth layout, as opposed to prior work. This paper also proposes an implementation and benchmark of the framework to study its sensitivity to certain deep learning-related conditions. As the field is novel, and many questions remain to be answered, we do not focus on finding the most optimal implementation of the method, but rather contribute toward a better understanding of the approach potential. More precisely, we study different learning strategies relative to the models training datasets. Finally, we discuss the main advantages and limitations of DL4GD.

In 2D visualizations, visibility of every datum's representation is crucial to ease the completion of visual tasks. Such a guarantee is barely respected in complex visualizations, mainly because of overdraws between datum representations that hide parts of the information (e.g., outliers). The literature proposes various Layout Adjustment algorithms to improve the readability of visualizations that suffer from this issue. Manipulating the data in high-dimensional, geometric or visual space; they rely on different strategies with their own strengths and weaknesses. Moreover, most of these algorithms are computationally expensive as they search for an exact solution in the geometric space and do not scale well to large datasets. This article proposes GIST, a layout adjustment algorithm that aims at optimizing three criteria: (i) node visibility guarantee (at least 1 pixel), (ii) node size maximization, and (iii) the original layout preservation. This is achieved by combining a search for the maximum node size that enables to draw all the data points without overlaps, with a limited budget of movements (i.e., limiting the distortions of the original layout). The method's basis relies on the idea that it is not necessary for two data representations to be strictly not overlapping in order to guarantee their visibility in visual space. Our algorithm therefore uses a tolerance in the geometric space to determine the overlaps between pairs of data. The tolerance is optimized such that the approximation computed in the geometric space can lead to visualization without noticeable overdraw after the data rendering rasterization. In addition, such an approximation helps to ease the algorithm's convergence as it reduces the number of constraints to resolve, enabling it to handle large datasets. We demonstrate the effectiveness of our approach by comparing its results to those of state-of-the-art methods on several large datasets.

In many 2D visualizations, data points are projected without considering their surface area, although they are often represented as shapes in visualization tools. These shapes support the display of information such as labels or encode data with size or color. However, inappropriate shape and size selections can lead to overlaps that obscure information and hinder the visualization's exploration. Overlap Removal (OR) algorithms have been developed as a layout post-processing solution to ensure that the visible graphical elements accurately represent the underlying data. As the original data layout contains vital information about its topology, it is essential for OR algorithms to preserve it as much as possible. This article presents an extension of the previously published FORBID algorithm by introducing a new approach that models OR as a joint stress and scaling optimization problem, utilizing efficient stochastic gradient descent. The goal is to produce an overlap-free layout that proposes a compromise between compactness (to ensure the encoded data is still readable) and preservation of the original layout (to preserve the structures that convey information about the data). Additionally, this article proposes SORDID, a shape-aware adaptation of FORBID that can handle the OR task on data points having any polygonal shape. Our approaches are compared against state-of-the-art algorithms, and several quality metrics demonstrate their effectiveness in removing overlaps while retaining the compactness and structures of the input layouts.

The rise of Deep Learning (DL) has led to a breakthrough in the research field of content-based multimedia indexing. Newly developed systems based on complex models outperform classic machine learning algorithms in object detection, image segmentation or classification tasks. However, despite their high performance, these systems still make mistakes. To be used in industrial conditions, these systems must be able to provide trustworthy decisions with guarantees or justifications. Therefore, it is crucial to provide means to analyze and comprehend the decision process that leads a model to its decision. Image classification implies tracking and understanding which input features the model relies on to make its prediction. This paper focuses on features attribution techniques and proposes Heatmaps by Hierarchical Occlusion (H2O), a novel method for detecting pattern-relevant features in an image. We also propose two new pairs of metrics that overcome some evaluation issues: (a) Insertion and Deletion Spearman correlation coefficients which both estimate a correlation between the computed scores in a saliency map and the importance for the model of the associated pixels in the image. (b) Insertion Positive and Deletion Negative Gradient Sums both estimate the coherence of the scores in the saliency maps. Both visual inspection and evaluation on 7 metrics show that H2O is competitive against state-of-the-art methods.

In the field of Explainable AI, multiples evaluation metrics have been proposed in order to assess the quality of explanation methods w.r.t. a set of desired properties. In this work, we study the articulation between the stability, correctness and plausibility of explanations based on feature importance for image classifiers. We show that the existing metrics for evaluating these properties do not always agree, raising the issue of what constitutes a good evaluation metric for explanations. Finally, in the particular case of stability and correctness, we show the possible limitations of some evaluation metrics and propose new ones that take into account the local behaviour of the model under test.

Abstract The use and creation of machine‐learning‐based solutions to solve problems or reduce their computational costs are becoming increasingly widespread in many domains. Deep Learning plays a large part in this growth. However, it has drawbacks such as a lack of explainability and behaving as a black‐box model. During the last few years, Visual Analytics has provided several proposals to cope with these drawbacks, supporting the emerging eXplainable Deep Learning field. This survey aims to (i) systematically report the contributions of Visual Analytics for eXplainable Deep Learning; (ii) spot gaps and challenges; (iii) serve as an anthology of visual analytical solutions ready to be exploited and put into operation by the Deep Learning community (architects, trainers and end users) and (iv) prove the degree of maturity, ease of integration and results for specific domains. The survey concludes by identifying future research challenges and bridging activities that are helpful to strengthen the role of Visual Analytics as effective support for eXplainable Deep Learning and to foster the adoption of Visual Analytics solutions in the eXplainable Deep Learning community. An interactive explorable version of this survey is available online at https://aware‐diag‐sapienza.github.io/VA4XDL .

The world is facing a new era in which social media communication plays a fundamental role in people’s lives. Along with proven benefits, several collateral drawbacks have risen, one being the widespread of false information with malicious intents, oftentimes using anonymous or false identities. Fighting this problem is challenging, especially when considering the nature of text messages involved on social media platforms: a sea of small messages and a myriad of users. Attributing the authorship of such messages is an ambitious endeavor; nevertheless, it is a way to fight this undesired disinformation scenario. In this work, we tackle the problem of authorship attribution of tiny messages, but, different from what has been done with longer texts, we rely upon data-driven approaches, avoiding handcraft features and harnessing recent advances of deep neural networks in the field of pattern recognition. By modeling small texts employed in social media as unidimensional signals, we propose a deep learning model to project these messages onto a manifold suitable for the task of authorship attribution. We provide two state-of-the-art solutions tailored for different setups and strategies for the scenario of authorship verification. These advances were possible, thanks to three additional contributions: an updated dataset based on the Twitter® platform, new sanitization techniques to improve the quality of the training data, and novel visual analytics techniques to help the development of authorship attribution solutions.

Current deep learning approaches are cutting-edge methods for solving classification tasks. Arising transfer learning techniques allows applying large generic model to simple tasks whereas simpler models could be used. Large models raise the major problem of their memory consumption and processor usage and lead to a prohibitive ecological footprint. In that paper, we present a novel visual analytics approach to interactively prune those networks and thus limit that issue. Our technique leverages a novel sparkline matrix visualization technique as well as a novel local metric which evaluates the discriminatory power of a filter to guide the pruning process and make it interpretable. We assess the well- founded of our approach through two realistic case studies and a user study. For both of them, the interactive refinement of the model led to a significantly smaller model having similar prediction accuracy than the original one.

Projecting a set of$n$points on a grid of size$\sqrt{n}\times\sqrt{n}$provides the best possible information density in two dimensions without overlap. We leverage the Voronoi Relaxation method to devise a novel and versatile post-processing algorithm called VRGrid: it enables the arrangement of any 2D data on a grid while preserving its initial positions. We apply VRGrid to generate compact and overlap-free visualization of popular and overlap-prone projection methods (e.g., t-SNE). We prove that our method complexity is$O(\sqrt{n}.i.n.log(n))$, with i a determined maximum number of iterations and$n$the input dataset size. It is thus usable for visualization of several thousands of points. We evaluate VRGrid's efficiency with several metrics: distance preservation (DP), neighborhood preservation (NP), pairwise relative positioning preservation (RPP) and global positioning preservation (GPP). We benchmark VRGrid against two state-of-the-art methods: Self-Sorting Maps (SSM) and Distance-preserving Grid (DGrid). VRGrid outperforms these two methods, given enough iterations, on DP, RPP and GPP which we identify to be the key metrics to preserve the positions of the original set of points.

Current machine learning and deep learning approaches are cutting-edge methods for solving classification tasks. Comparing the performances of classification models has become a prominent task since the outbreak of these techniques. The performance of such classification models is measured by the ratio between the correctly predicted samples and the others. The most widely used visualization to represent this information is the Confusion matrix. Yet, if this technique is suited to apprehend one model performances, very few works use this representation to compare models. In that paper, we present the Relative Confusion Matrix (RCM), a new matrix visualization that leverages Confusion matrices and a color encoding to expose the class-wise differences of performances between two models. We conduct a user evaluation to compare RCM with two confusion matrix variants. Our results show that RCM encoding leads to a more efficient comparison of two models than existing approaches.

The design of efficient representations is well established as a fruitful way to explore and analyze complex or large data. In these representations, data are encoded with various visual attributes depending on the needs of the representation itself. To make coherent design choices about visual attributes, the visual search field proposes guidelines based on the human brain’s perception of features. However, information visualization representations frequently need to depict more data than the amount these guidelines have been validated on. Since, the information visualization community has extended these guidelines to a wider parameter space. This paper contributes to this theme by extending visual search theories to an information visualization context. We consider a visual search task where subjects are asked to find an unknown outlier in a grid of randomly laid out distractors. Stimuli are defined by color and shape features for the purpose of visually encoding categorical data. The experimental protocol is made of a parameters space reduction step (i.e., sub-sampling) based on a machine learning model, and a user evaluation to validate hypotheses and measure capacity limits. The results show that the major difficulty factor is the number of visual attributes that are used to encode the outlier. When redundantly encoded, the display heterogeneity has no effect on the task. When encoded with one attribute, the difficulty depends on that attribute heterogeneity until its capacity limit (7 for color, 5 for shape) is reached. Finally, when encoded with two attributes simultaneously, performances drop drastically even with minor heterogeneity.

In information visualization, it has become mandatory to assess visualization techniques efficiency either to write a survey, optimize a technique or even design a new one. To do so, the common way is to conduct user evaluations through which human subjects are asked to solve a task on different visualization techniques while their performances are measured to assess which technique is the most efficient. These evaluations can be complex to design and setup in order not to be biased and, in the end, their results can become contestable when the evaluation methods standards evolve. To overcome these flaws, new evaluation methods are emerging, mostly making use of modern and efficient computer vision techniques such as deep learning. These new methods rely on a strong assumption that has not been studied deeply enough yet: humans and deep learning models performances can be correlated. This paper explores the performances of both a state-of-the-art deep neural network and human subjects on an outlier detection task taken from a previous experiment of the literature. The objective is to study whether the machine and humans behaviors were different or if some correlations can be observed. Our study shows that their results are significantly correlated and a machine learning model efficiently learned to predict human performances using deep neural network metrics as input. Hence, this work presents a use case where using a deep neural network to assess human subjects performances is efficient.

Many end-user evaluations of data visualization techniques have been run during the last decades. Their results are cornerstones to build efficient visualization systems. However, designing such an evaluation is always complex and time-consuming and may end in a lack of statistical evidence and reproducibility. We believe that modern and efficient computer vision techniques, such as deep convolutional neural networks (CNNs), may help visualization researchers to build and/or adjust their evaluation hypothesis. The basis of our idea is to train machine learning models on several visualization techniques to solve a specific task. Our assumption is that it is possible to compare the efficiency of visualization techniques based on the performance of their corresponding model. As current machine learning models are not able to strictly reflect human capabilities, including their imperfections, such results should be interpreted with caution. However, we think that using machine learning-based pre-evaluation, as a pre-process of standard user evaluations, should help researchers to perform a more exhaustive study of their design space. Thus, it should improve their final user evaluation by providing it better test cases. In this paper, we present the results of two experiments we have conducted to assess how correlated the performance of users and computer vision techniques can be. That study compares two mainstream graph visualization techniques: node-link (NL) and adjacency-matrix (AM) diagrams. Using two well-known deep convolutional neural networks, we partially reproduced user evaluations from Ghoniem et al. and from Okoe et al.. These experiments showed that some user evaluation results can be reproduced automatically.

Biometric authentication systems verify the identity of individuals based on what they are. As they are error prone, they can reject genuine individuals or accept impostors. Researchers of the field quantify the quality of their algorithm by benchmarking it on several databases. However, although the standard evaluation metrics state the performance of their system, they are unable to explain the reasons of their errors. This paper presents a novel way to visualize the evaluation results of a biometric authentication system which helps to find which individuals or samples are sources of errors. This knowledge could help to fix the algorithms. A biometric database of scores is modeled as a partitioned power-graph with nodes representing biometric samples and power-nodes representing individuals. A novel recursive edge bundling method is also applied to reduce clutter. This proposal has been successfully applied on several biometric databases and has proved its efficiency.

Biometric authentication systems suffer from several performance limitations. Many performance metrics exist to assess the overall performance of such systems. However, these metrics provide a quantitative assessment in terms of errors without explaining the reasons behind the set of users who significantly contributed for these errors. Towards contributing to solve this problem, we present a novel method (named Zoo Graph) to visualize the performance of a biometric system as a graph thanks to a database of recognition scores. Our approach is an improvement of the Zoo Plot and emphasizes on the relations between the individuals of the database and allows interactive manipulations to track these relations and understand why the biometric authentication method reacts this way. This graph provides researchers with an additional visual assessment tool that would identify problematic users. Such information would allow researchers to update their developed authentication algorithms to reduce those errors.

International audience

The context of this paper is the labelling of a document image database in an industrial process. Our work focuses on the quality assessment of a given labelled database. In most practical cases, a database is manually labelled by an operator who has to browse sequentially the images (presented as thumbnails) until the whole database is labelled. This task is very repetitive; moreover the filing plan defining the names and number of classes is often incomplete, which leads to many labelling errors. The question is then to certify if the quality of a labelled batch is good enough to globally accept it. Our objective is to ease and speed up that evaluation that needs up to 1.5 more times than the labelling work itself. We propose an interactive tool for visualizing the data as a graph. That graph enhances similarities between documents as well as the labelling quality. We define criteria on the graph that characterize the three types of errors an operator can do: an image is mislabelled, one class should be split in more pertinent subclasses, several classes should be merged in another. This allows us to focus the operator attention on potential errors. He can then count the errors encountered while auditing the database and assess (or not) the global labelling quality.

Graph is a powerful tool to model relationships between elements and has been widely used in different research areas. Size and complexity of newly acquired graphs prohibit manual representations and urge a need for automatic visualization methods. We are interested with the node-links diagram which represents each node as a glyph and edge as a line between the corresponding nodes. % We present a novel layout algorithm that emphasizes the cores of very large networks (up to several hundred thousand of nodes and million of edges) in few seconds or minutes. Our method uses a hierarchical coreness decomposition of the graph and a combination of existing layout algorithms according to the clusters topologies. Area-aware drawing algorithms which produce node overlap-free drawings are used to reduce the visual clutter. Edges are bundled along the hierarchy of clusters to highlight the network communities and reduce edge visual clutter. % We validated our approach by comparing our method against one of the fastest method of the state of the art on a benchmark of 23 large graphs extracted from various sources. We have statistically proved that our method performs faster while providing meaningful results.

Generally, fraud risk implies any intentional deception made for financial gain. In this paper, we consider this risk in the field of services which support transactions with electronic money. Specifically, we apply a tool for predictive security analysis at runtime which observes process behavior with respect to transactions within a money transfer service and tries to match it with expected behavior given by a process model. We analyze deviations from the given behavior specification for anomalies that indicate a possible misuse of the service related to money laundering activities. We evaluate the applicability of the proposed approach and provide measurements on computational and recognition performance of the tool - Predictive Security Analyser - produced using real operational and simulated logs. The goal of the experiments is to detect misuse patterns reflecting a given money laundering scheme in synthetic process behavior based on properties captured from real world transaction events.

We propose in this paper a low cost multimodal biometric system combining keystroke dynamics and 2D face recognition. The objective of the proposed system is to be used while keeping in mind: good performances, acceptability, and espect of privacy. Different fusion methods have been used (min, max, mul, svm, weighted sum configured with genetic algorithms, and, genetic programming) on the scores of three keystroke dynamics algorithms and two 2D face recognition ones. This multimodal biometric system improves the recognition rate in comparison with each individual method. On a chimeric database composed of 100 individuals, the best keystroke dynamics method obtains an EER of 8.77%, the best face recognition one has an EER of 6.38%, while the best proposed fusion system provides an EER of 2.22%.