Recent studies have shown that adversarial examples in state-of-the-art image classifiers trained by deep neural networks (DNN) can be easily generated when the target model is transparent to an attacker, known as the white-box setting. However, when attacking a deployed machine learning service, one can only acquire the input-output correspondences of the target model; this is the so-called black-box attack setting. The major drawback of existing black-box attacks is the need for excessive model queries, which may give a false sense of model robustness due to inefficient query designs. To bridge this gap, we propose a generic framework for query-efficient blackbox attacks. Our framework, AutoZOOM, which is short for Autoencoder-based Zeroth Order Optimization Method, has two novel building blocks towards efficient black-box attacks: (i) an adaptive random gradient estimation strategy to balance query counts and distortion, and (ii) an autoencoder that is either trained offline with unlabeled data or a bilinear resizing operation for attack acceleration. Experimental results suggest that, by applying AutoZOOM to a state-of-the-art black-box attack (ZOO), a significant reduction in model queries can be achieved without sacrificing the attack success rate and the visual quality of the resulting adversarial examples. In particular, when compared to the standard ZOO method, AutoZOOM can consistently reduce the mean query counts in finding successful adversarial examples (or reaching the same distortion level) by at least 93% on MNIST, CIFAR-10 and ImageNet datasets, leading to novel insights on adversarial robustness.

Stochastic computing (SC) computes with randomized bit-streams using standard logic circuits. Its defining features are low power, small area, and high fault tolerance; its drawbacks are long run times and inaccuracies due to its inherently random behavior. Consequently, much previous work has focused on improving SC performance by introducing non-random or deterministic data formats and components, often at considerable cost. However, as this paper shows, taking advantage of, or even adding to, a stochastic circuit's randomness can play a major positive role in applications like neural networks (NNs). The amount of such randomness, must however, be carefully controlled to achieve a beneficial effect without corrupting an application's functionality. The paper first discusses the use of mean square deviation (MSD) as a metric for randomness in SC. It then describes a low-cost element to control the MSD levels of stochastic signals. Finally, it examines two applications where SC can provide performance-enhancing randomness at very low cost, while retaining all the other benefits of SC. Specifically, it is shown how to improve the visual quality of black-and-white images via stochastic dithering, a technique that leverages randomness to enhance image details. Further, the paper demonstrates how the randomness of an SC-based layer makes an NN more resilient against adversarial attacks than an NN realized entirely by conventional, non-stochastic designs.

Stochastic computing (SC) is an alternative way to design arithmetic circuits that have lower area and power than circuits employing conventional binary (base-2) computing (BC). Large SC-based systems like neural networks (NNs) or vision systems usually resort to BC, either explicitly or implicitly, for tasks requiring higher accuracy, such as data storage, control functions, or complex arithmetic operations. The resulting hybrid SC-BC features often incorporate unsatisfactory tradeoffs between system latency and accuracy. For example, they may require many costly SC-BC data format converters, which interrupt bit-stream flow and cause significant delay overhead. While improving accuracy has been a major research goal in SC, less attention has been paid to reducing latency. We present a novel design methodology called Maxflow that minimizes the latency of SC operations without reducing accuracy or interrupting data flow more than necessary. Maxflow supports delay-accuracy tradeoffs with little hardware modification. Its effectiveness is demonstrated for a deep NN, where it reduces overall latency substantially with no loss of accuracy compared to previous designs.

As application demands for zeroth-order (gradient-free) optimization accelerate, the need for variance reduced and faster converging approaches is also intensifying. This paper addresses these challenges by presenting: a) a comprehensive theoretical analysis of variance reduced zeroth-order (ZO) optimization, b) a novel variance reduced ZO algorithm, called ZO-SVRG, and c) an experimental evaluation of our approach in the context of two compelling applications, black-box chemical material classification and generation of adversarial examples from black-box deep neural network models. Our theoretical analysis uncovers an essential difficulty in the analysis of ZO-SVRG: the unbiased assumption on gradient estimates no longer holds. We prove that compared to its first-order counterpart, ZO-SVRG with a two-point random gradient estimator could suffer an additional error of order $O(1/b)$, where $b$ is the mini-batch size. To mitigate this error, we propose two accelerated versions of ZO-SVRG utilizing variance reduced gradient estimators, which achieve the best rate known for ZO stochastic optimization (in terms of iterations). Our extensive experimental results show that our approaches outperform other state-of-the-art ZO algorithms, and strike a balance between the convergence rate and the function query complexity.

In this paper we propose a novel method that provides contrastive explanations justifying the classification of an input by a black box classifier such as a deep neural network. Given an input we find what should be minimally and sufficiently present (viz. important object pixels in an image) to justify its classification and analogously what should be minimally and necessarily \emph{absent} (viz. certain background pixels). We argue that such explanations are natural for humans and are used commonly in domains such as health care and criminology. What is minimally but critically \emph{absent} is an important part of an explanation, which to the best of our knowledge, has not been explicitly identified by current explanation methods that explain predictions of neural networks. We validate our approach on three real datasets obtained from diverse domains; namely, a handwritten digits dataset MNIST, a large procurement fraud dataset and a brain activity strength dataset. In all three cases, we witness the power of our approach in generating precise explanations that are also easy for human experts to understand and evaluate.

The success of the application of machine-learning techniques to compilation tasks can be largely attributed to the recent development and advancement of program characterization, a process that numerically or structurally quantifies a target program. While great achievements have been made in identifying key features to characterize programs, choosing a correct set of features for a specific compiler task remains an ad hoc procedure. In order to guarantee a comprehensive coverage of features, compiler engineers usually need to select excessive number of features. This, unfortunately, would potentially lead to a selection of multiple similar features, which in turn could create a new problem of bias that emphasizes certain aspects of a program's characteristics, hence reducing the accuracy and performance of the target compiler task. In this paper, we propose FEAture Selection for compilation Tasks (FEAST), an efficient and automated framework for determining the most relevant and representative features from a feature pool. Specifically, FEAST utilizes widely used statistics and machine-learning tools, including LASSO, sequential forward and backward selection, for automatic feature selection, and can in general be applied to any numerical feature set. This paper further proposes an automated approach to compiler parameter assignment for assessing the performance of FEAST. Intensive experimental results demonstrate that, under the compiler parameter assignment task, FEAST can achieve comparable results with about 18% of features that are automatically selected from the entire feature pool. We also inspect these selected features and discuss their roles in program execution.

Interest in stochastic computing (SC) has been growing due to the need for low-cost circuitry in areas like image processing and machine learning. While combinational stochastic circuits are relatively easy to design, their limitations such as having few implementable functions and inaccuracies due to correlation, call for sequential components and design methods. The theory underlying sequential stochastic circuits is not fully understood, however. Almost all existing sequential SC circuits fall into the up/down-counter-based (UCB) class. In this paper, we identify and investigate a new SC circuit class: shift-register-based (SRB). We focus on the properties of SRB circuits and demonstrate their central role in stochastic computing. This leads to an algorithm MOUSE for sequential design optimization. By exploiting sequential stochastic equivalence, MOUSE can reduce SRB circuit cost without compromising performance or accuracy.

Stochastic computing (SC) performs arithmetic on randomized bit-streams called stochastic numbers (SNs) using standard logic circuits. SC has many appealing features such as error tolerance, low power, and low area cost. However, it suffers from severe accuracy loss due to correlation or insufficient randomness. SNs can be decorrelated by regenerating them from independent random sources. This is the preferred decorrelation method mentioned in the literature, but it often entails huge area and delay overhead. An attractive alternative is isolation-based decorrelation, which is the focus of this research. Isolation works by inserting delays (isolators) into a stochastic circuit to eliminate undesirable interactions among its SNs. Surprisingly, although it has far lower cost than regeneration, isolation has not been studied systematically before, hindering its practical use. The paper first examines the basic characteristics of SC isolation. We show that unless carefully used, it can result in excessive isolator numbers or unexpectedly corrupt a circuit's function. We therefore formally characterize the behavior of an isolation-decorrelated circuit, and derive conditions for correct deployment of isolators. We then describe the first isolator placement algorithm designed to minimize the number of isolators. Finally, we present supporting data obtained from simulation experiments on representative circuits.

Stochastic computing (SC) is a re-emerging technique to process probability data encoded in digital bit-streams. Its main advantage is that arithmetic operations can be implemented by extremely small and low-power logic circuits. This makes SC suitable for signal-processing applications involving matrix operations whose VLSI implementation is very costly. Previous SC approaches only address basic matrix operations with relatively low accuracy needs. We explore the use of SC to implement a representative complex matrix operation, namely eigenvector computation. We apply it to a training task for visual face recognition, and show that our SC design has performance comparable to its conventional binary counterpart, while being able to trade computation time for accuracy.

Thread divergence frequently occurs in various GPGPU applications while parallel threads encounter branching, especially for multimedia algorithms involving high-level image processing and computer vision algorithms. Either predicate or branch instruction degrades the parallel processing performance as a result of synchronization cost. In this work, we propose a configurable thread compaction unit (TCU) to relieve such execution overhead. Through early-stage compacting divergent threads by evaluating a compacting function with a compacting map, TCU can prevent redundant executions caused by predicate instruction, or repetitively invoking processors to fetch instructions and validate effective branches. Our simulation results show that, with TCU, GPUs can improve up to 24.5x and 1.8x performance for Viola-Jones face detection framework compared to predicate and branch instruction. Furthermore, 4.3x and 1.4x improvement in salient region linear feature extraction can be achieved as well. Finally, cache issue in TCU architecture is also discussed in detail.