With the large-scale integration and use of neural network models, especially in critical embedded systems, their security assessment to guarantee their reliability is becoming an urgent need. More particularly, models deployed in embed-ded platforms, such as 32-bit microcontrollers, are physically accessible by adversaries and therefore vulnerable to hardware disturbances. We present the first set of experiments on the use of two fault injection means, electromagnetic and laser injections, applied on neural networks models embedded on a Cortex M4 32-bit microcontroller platform. Contrary to most of state-of-the-art works dedicated to the alteration of the internal parameters or input values, our goal is to simulate and experimentally demonstrate the impact of a specific fault model that is instruction skip. For that purpose, we assessed several modification attacks on the control flow of a neural network inference. We reveal integrity threats by targeting several steps in the inference program of typical convolutional neural network models, which may be exploited by an attacker to alter the predictions of the target models with different adversarial goals.

Deep neural network models are massively deployed on a wide variety of hardware platforms. This results in the appearance of new attack vectors that significantly extend the standard attack surface, extensively studied by the adversarial machine learning community. One of the first attack that aims at drastically dropping the performance of a model by targeting its parameters stored in memory, is the Bit-Flip Attack (BFA). In this work, we point out several evaluation challenges related to the BFA. First, the lack of an adversary’s budget in the standard threat model is problematic, especially when dealing with physical attacks. Moreover, since the BFA presents critical variability, we discuss the influence of some training parameters and the importance of the model architecture. This work is the first to present the impact of the BFA against fully-connected architectures that present different behaviors compared to convolutional neural networks. These results highlight the importance of defining robust and sound evaluation methodologies to properly evaluate the dangers of parameter-based attacks as well as measure the real level of robustness offered by a defense.

The vulnerability of Deep Neural Network (DNN) models to maliciously crafted adversarial perturbations is a critical topic considering their ongoing large-scale deployment. In this work, we explore an interesting phenomenon that occurs when an image is reinjected multiple times into a DNN, according to a procedure (called reverberation) that has been first proposed in cognitive psychology to avoid the catastrophic forgetting issue, through its impact on adversarial perturbations. We describe reverberation in vanilla autoencoders and propose a new reverberant architecture combining a classifier and an autoencoder that allows the joint observation of the logits and reconstructed images. We experimentally measure the impact of reverberation on adversarial perturbations placing ourselves in a scenario of adversarial example detection. The results show that clean and adversarial examples – even with small levels of perturbation – behave very differently throughout reverberation. While computationally efficient (reverberation is only based on inferences), our approach yields promising results for adversarial examples detection, consistent across datasets, adversarial attacks and DNN architectures.

The growing interest for adversarial examples, i.e. maliciously modified examples which fool a classifier, has resulted in many defenses intended to detect them, render them inoffensive or make the model more robust against them. In this paper, we pave the way towards a new approach to improve the robustness of a model against black-box transfer attacks. A removable additional neural network is included in the target model, and is designed to induce the luring effect, which tricks the adversary into choosing false directions to fool the target model. Training the additional model is achieved thanks to a loss function acting on the logits sequence order. Our deception-based method only needs to have access to the predictions of the target model and does not require a labeled data set. We explain the luring effect thanks to the notion of robust and non-robust useful features and perform experiments on MNIST, SVHN and CIFAR10 to characterize and evaluate this phenomenon. Additionally, we scale the luring effect to ImageNet, experiment practical use of it and discuss its complementarity with other defense schemes.

Adversarial examples mainly exploit changes to input pixels to which humans are not sensitive to, and arise from the fact that models make decisions based on uninterpretable features. Interestingly, cognitive science reports that the process of interpretability for human classification decision relies predominantly on low spatial frequency components. In this paper, we investigate the robustness to adversarial perturbations of models enforced during training to leverage information corresponding to different spatial frequency ranges. We show that it is tightly linked to the spatial frequency characteristics of the data at stake. Indeed, depending on the data set, the same constraint may results in very different level of robustness (up to 0.41 adversarial accuracy difference). To explain this phenomenon, we conduct several experiments to enlighten influential factors such as the level of sensitivity to high frequencies, and the transferability of adversarial perturbations between original and low-pass filtered inputs.

Laser injection is a powerful fault injection technique with a high spatial accuracy which allows an adversary to efficiently extract the secret information from an electronic device. The control and the repeatability of faults requires the attacker to understand the relation of the fault model to the setup (notably the laser spot size) and the process node of the target device. Most studies on laser fault injection report fault models resulting from a photo-electric current in CMOS transistors. This study provides a black-box analysis of the effect of a photo-electric current in floating-gate transistors of two embedded NOR Flash memories from two different manufacturers. Experimental results demonstrate that single-bit bit-set faults can be injected in code and data without corrupting the Flash memory, even with a laser spot of more than 20 μm in diameter, which is several orders of magnitude larger than the process node of the floating-gate transistors in the experiments. This article also presents the specifics of performing a "safe-error" attack on AES, leveraging the previously detailed single-bit bit-set fault model.

As the will to deploy neural network models on embedded systems grows, and considering the related memory footprint and energy consumption requirements, finding lighter solutions to store neural networks such as parameter quantization and more efficient inference methods becomes major research topics. Parallel to that, adversarial machine learning has risen recently, unveiling some critical flaws of machine learning models, especially neural networks. In particular, perturbed inputs called adversarial examples have been shown to fool a model into making incorrect predictions. In this paper, we investigate the adversarial robustness of quantized neural networks under different attacks. We show that quantization is not a robust protection when considering advanced threats and may result in severe form of gradient masking which leads to a false impression of security. However, and interestingly, we experimentally observe poor transferability capacities between full-precision and quantized models and between models with different quantization levels which we explain by the quantization value shift phenomenon and gradient misalignment.

Tourist photographs constitute a large part of the images uploaded to photo sharing platforms. But filtering methods are needed before one can extract useful knowledge from noisy user-supplied metadata. Here we show how to extract clean trip related information (what people visit, for how long, panoramic spots) from Flickr metadata. We illustrate our technique on a sample of metadata and images covering 183 cities of different size and from different parts of the world.

Web image search is inspired by text search techniques; it mainly relies on indexing textual data that surround the image file. But retrieval results are often noisy and image processing techniques have been proposed to rerank images. Unfortunately, these techniques usually imply a computational overload that makes the reranking process intractable in real time. We introduce here a lightweight reranking method that compares each result not only to the other query results but also to an external, contrastive class of items. The external class contains diversified images; the intuition supporting our approach is that results that are visually similar to other query results but dissimilar to elements of the contrastive class are likely to be good answers. The success of visual reranking depends on the visual coherence of queries; we measure this coherence in order to evaluate the chances of success. Visual reranking tends to emerge near duplicate images and we complement it with a diversification function which ensures that different aspects of a query are presented to the user. Our method is evaluated against a standard search engine using 210 diversified queries. Significant improvements are reported for both quantitative and qualitative tests.

In this paper we describe Olive, a concept based image search engine implemented using the WordNet nouns hierarchy. The system reformulates textual queries and performs an on the fly search for Google images corresponding to leaf nodes that are found under the currently demanded term. The retrieved pictures are rendered in a conceptually structured fashion and semantically related queries are presented to the user. In addition, a content based search in conceptually controlled neighborhoods is proposed.