Deep Neural Networks have emerged as state-of-the-art solutions for complex intelligence problems. DNNs derive their predictive power by learning from millions of training examples in either a supervised or semi-supervised fashion. As such, a critical aspect of the DNN system design procedure is the collection of large annotated training datasets that exhibit high coverage of the problem space. While data synthesis and annotation techniques have been proposed to mitigate the burden of acquiring large datasets, these methods do not quantify the usefulness of each generated dataset and its subsequent impact on training effort. In this work we establish parallels between the autonomous design of DNNs for machine vision applications and the task of functionally verifying a hardware design. Similar to automatic test vector generation, we propose a technique that progressively generates training datasets using virtual synthetic models. Furthermore, we propose an automated DNN design framework that jointly tries to stochastically maximize training coverage while minimizing the number of training and validation cycles utilizing insights from functional verification.

As a benefactor of the proliferation of large scale integration, traditionally simple and unintelligent devices such as cameras have been transformed into key components of rich and engaging smart environments. By integrating machine perception algorithms, these cognitive cameras have the ability to perceive and understand their environments. A principal barrier to realizing the potential of cognitive cameras has been the absence of sufficient computing power within the device. This is especially true in wearable devices which are limited by both compute capability and energy. Hardware customization and specialization present effective solutions to the power and performance bottlenecks that have limited ubiquitous adoption. This work details the architecture, design, and evaluation of a cognitive camera system that employs custom hardware to meet both power and performance constraints. Furthermore we illustrate its use as an assistance system for the visually impaired.

As high-fidelity small form-factor cameras become increasingly available and affordable, there will be a subsequent growth and emergence of vision-based applications that take advantage of this increase in visual information. The key challenge is for the embedded systems, on which the bulk of these applications will be deployed, to maintain real-time performance in the midst of the exponential increase in spatial and temporal visual data. For example, a useful vision-based driver assistance system needs to locate and identify critical objects such as pedestrians, other vehicles, pot-holes, animals, and street signs with latency small enough to allow a human driver to react accordingly. In this work, we propose a digital accelerator architecture for a high-throughput, robust, scalable, and tunable visual object detection pipeline based on Histogram of Oriented Gradients (HOG) features. From a systems perspective, efficacy can be measured in terms of speed, accuracy, energy efficiency and scalability in performing such visual tasks. Since each application dictates the criticality of any one of these dimensions, our proposed architecture exposes design-time parameters that can take advantage of domain-specific knowledge while supporting tune-ability through run-time configurations. To evaluate the effectiveness of our vision accelerator we map the architecture to a modern FPGA and demonstrate full HD video processing at 30 fps (frames per second) operating at a conservative 100 MHz clock. Evaluations on a single object class show throughput improvements of 2× and 5× over GPU and multi-threaded CPU implementations respectively. Further more we provide a pathway for enhanced scalability for the many-class problem and achieve over 20× improvement over an equivalent CPU implementation for 5 object classes.

Limited power budgets and the need for high performance computing have led to platform customization with a number of accelerators integrated with CMPs. In order to study customized architectures, we model four customization design points and compare their performance and energy across a number of computer vision workloads. We analyze the limitations of generic architectures and quantify the costs of increasing customization using these micro-architectural design points. This analysis leads us to develop a framework consisting of low-power multi-cores and an array of configurable micro-accelerator functional units. Using this platform, we illustrate dataflow and control processing optimizations that provide for performance gains similar to custom ASICs for a wide range of vision benchmarks.

Recently, biologically inspired vision systems have been the focus of intense research effort to emulate the high energy-efficiency, performance and robustness of mammalian vision systems. However, previous vision accelerators have only focused on speeding up computationally intense portions of the system without exploiting effects seen in the human brain that demonstrate the task influence in the vision mechanism. In this paper, we propose a task-oriented two-level vision system which is composed of Saliency and SURF. To the best of our knowledge, our design is the first embedded system that utilizes task influence in the computation of visual attention and recognition. As a result, we show that the new system can achieve at most 12.75% accuracy improvement while saving 25% computation work.

Embedded visual assist systems are emerging as increasingly viable tools for aiding visually impaired persons in their day-to-day life activities. Novel wearable devices with imaging capabilities will be uniquely positioned to assist visually impaired in activities such as grocery shopping. However, supporting such time-sensitive applications on embedded platforms requires an intelligent trade-off between accuracy and computational efficiency. In order to maximize their utility in real-world scenarios, visual classifiers often need to recognize objects within large sets of object classes that are both diverse and deep. In a grocery market, simultaneously recognizing the appearance of people, shopping carts, and pasta is an example of a common diverse object classification task. Moreover, a useful visual-aid system would need deep classification capability to distinguish among the many styles and brands of pasta to direct attention to a particular box. Exemplar Support Vector Machines (ESVMs) provide a means of achieving this specificity, but are resource intensive as computation increases rapidly with the number of classes to be recognized. To maintain scalability without sacrificing accuracy, we examine the use of a biologically-inspired classifier (HMAX) as a front-end filter that can narrow the set of ESVMs to be evaluated. We show that a hierarchical classifier combining HMAX and ESVM performs better than either of the two individually. We achieve 12% improvement in accuracy over HMAX and 4% improvement over ESVM while reducing computational overhead of evaluating all possible exemplars.

Video applications are becoming ubiquitous in mobile and embedded systems. Wearable video systems such as Google Glasses require capabilities for real-time video analytics and prolonged battery lifetimes. Further, the increasing resolution of image sensors in these mobile systems places an increasing demand on both the memory storage as well as the computational power. In this work, we present the Refresh Enabled Video Analytics (REVA) system, an embedded architecture for multi-object scene understanding and tackle the unique opportunities provided by real-time embedded video analytics applications for reducing the DRAM memory refresh energy. We compare our design with the existing design space and show savings of 88% in refresh power and 15% in total power, as compared to a standard DRAM refresh scheme.

This work proposes an adaptive architecture for a neuro-inspired vision pipeline that uses a data driven approach to dynamic configuration and optimization of the underlying computational fabric. The micro-architecture allows the adaptation mechanism to be tuned for changing system objectives that cross-cut accuracy, power consumption, and throughput constraints.

In this paper, a bio-inspired technique of finding the regions of highest visual importance within an image is proposed for reducing power consumption in modern liquid crystal displays (LCDs) that utilize a 2D light-emitting diode (LED) backlighting system. The conspicuity map generated from this neuromorphic saliency model, along with an adaptive dimming method, is applied to the backlighting array to reduce the luminance of regions of least interest as perceived by a human viewer. Corresponding image compensation is applied to the saliency modulated image to minimize distortion and retain the original image quality. Experimental results shows average 65% power can be saved when the original display system is integrated with a low-overhead real-time hardware implementation of the saliency model.

Summary form only given. The objective of this paper is to present a configurable architecture for a visual saliency model based on AIM. It presents algorithmic enhancements to AIM that facilitates the design of a performance-efficient hardware architecture that offers tradeoffs between accuracy, resource utilization and latency. The AIM computational model involves (1) extraction of a set of coefficient features for each local patch in an image, (2) estimation of probability density for each coefficient with respect to its local surround, (3) computation of their product to give a joint likelihood and (4) computation of the self information of each pixel from its log likelihood. Calculation of likelihood with respect to each pixel individually in a local surround is computationally expensive. It proposes to approximate the contribution of pixels in the surround in terms of “cells” grouped further into “support zones”, whose widths are configurable. This approximation leads to nearly a 10x reduction in the number of multipliers, a critical resource, for a 41x41 surround size.

The Human Visual System (HVS) exhibits multi-resolution characteristics, where the fovea is at the highest resolution while the resolution tapers off towards the periphery. Given enough activity at the periphery, the HVS is then capable to foveate to the next region of interest (ROI), to attend to it at full resolution. Saliency models in the past have focused on identifying features that can be used in a bottom-up manner to generate conspicuity maps, which are then combined together to provide regions of fixated interest. However, these models neglect to take into consideration the foveal relation of an object of interest. The model proposed in this work aims to compute saliency as a function of distance from a given fixation point, using a multi-resolution framework. Apart from computational benefits, significant motivation can be found from this work in areas such as visual search, robotics, communications etc.

In this paper we present Vortex: a reconfigurable Network-on-Chip platform suitable for implementing domain-specific hardware accelerators in a design efficient manner. Our Vortex platform provides a flexible means to compose domain-specific accelerators for streaming applications such as performance critical machine vision systems. By substituting a traditional shared-bus architecture with low latency packet-switched routers and high utility network adaptors, maximum performance is exploited with minimal regard to communication infrastructure design and validation. To highlight the utility of the Vortex platform we present a case study in which a video analytics pipeline is mapped onto a multi-FPGA system. The system meets real-time throughput requirements on 3 Megapixel 48-bit image sequences with minimal resource overhead attributed to the Vortex communication infrastructure.

Gridding is a method of interpolating irregularly sampled data on to a uniform grid and is a critical image reconstruction step in several applications which operate on non-Cartesian sampled data. In this paper, we present an algorithm architecture co-design framework for accelerating gridding using FPGAs. We present a parameterized hardware library for accelerating gridding to support both arbitrary and regular trajectories. We further describe our kernel automation framework which supports several kernel functions through look-up-table (LUT) based Taylor polynomial evaluation. This framework is integrated using an in-house multi-FPGA development platform which provides hardware infrastructure for integrating custom accelerators. Design-space exploration is enabled by an automation flow which allows system generation from an algorithm specification. We further provide several case studies by realizing systems for nonuniform fast Fourier transform (NuFFT) with different parameter sets and porting them on to the BEE3 platform. Results show speedups of more than 16X and 2X over existing CPU and FPGA implementations respectively, and up to 5.5 times higher performance-per-watt over a comparable GPU implementation.

Biological vision systems use saliency-based visual attention mechanisms to limit higher-level vision processing on the most visually-salient subsets of an input image. Among several computational models that capture the visual-saliency in biological system, an information theoretic AIM(Attention based on Information Maximization) algorithm has been demonstrated to predict human gaze patterns better than other existing models. We present an FPGA based implementation of this computationally intensive AIM algorithm to support embedded vision applications. Our implementation provides performance of processing about 4M pixels/sec for 25 basis functions with a convolution kernel size of 21 by 21 for each of the R, G, and B color-channels, when implemented on a Virtex-6 LX240T. We also provide an optimization aimed at controlling the trade-off between power consumption and latency, and performance comparisons with a GPU implementation.

Gabor filters are widely used to model the behavior of the V1 stage of the primary visual cortex of mammalian vision systems. Acceleration of this filter is critical for various real-time vision applications. In this work, we explore a variety of design options for the Gabor filter to tradeoff resource utilization, performance and accuracy. Our results indicate that our optimizations enable a variety of design options to meet a wide range of application constraints.

We present an FPGA accelerator for the Non-uniform Fast Fourier Transform, which is a technique to reconstruct images from arbitrarily sampled data. We accelerate the compute-intensive interpolation step of the NuFFT Gridding algorithm by implementing it on an FPGA. In order to ensure efficient memory performance, we present a novel FPGA implementation for Geometric Tiling based sorting of the arbitrary samples. The convolution is then performed by a novel Data Translation architecture which is composed of a multi-port local memory, dynamic coordinate-generator and a plug-and-play kernel pipeline. Our implementation is in single-precision floating point and has been ported onto the BEE3 platform. Experimental results show that our FPGA implementation can generate fairly high performance without sacrificing flexibility for various data-sizes and kernel functions. We demonstrate up to 8X speedup and up to 27 times higher performance-per-watt over a comparable CPU implementation and up to 20% higher performance-per-watt when compared to a relevant GPU implementation.

This paper describes TANOR, an automated framework for designing hardware accelerators for numerical computation on reconfigurable platforms. Applications utilizing numerical algorithms on large-size data sets require high-throughput computation platforms. The focus is on N-body interaction problems which have a wide range of applications spanning from astrophysics to molecular dynamics. The TANOR design flow starts with a MATLAB description of a particular interaction function, its parameters, and certain architectural constraints specified through a graphical user interface. Subsequently, TANOR automatically generates a configuration bitstream for a target FPGA along with associated drivers and control software necessary to direct the application from a host PC. Architectural exploration is facilitated through support for fully custom fixed-point and floating-point representations in addition to standard number representations such as single-precision floating point. Moreover, TANOR enables joint exploration of algorithmic and architectural variations in realizing efficient hardware accelerators. TANOR's capabilities have been demonstrated for three different N-body interaction applications: the calculation of gravitational potential in astrophysics, the diffusion or convolution with Gaussian kernel common in image processing applications, and the force calculation with vector-valued kernel function in molecular dynamics simulation. Experimental results show that TANOR-generated hardware accelerators achieve lower resource utilization without compromising numerical accuracy, in comparison to other existing custom accelerators.

In real-time video mining applications it is desirable to extract information about human subjects, such as gender, ethnicity, and age, from grayscale frontal face images. Many algorithms have been developed in the machine learning, statistical data mining, and pattern classification communities that perform such tasks with remarkable accuracy. Many of these algorithms, however, when implemented in software, suffer poor frame rates due to the amount and complexity of the computation involved. This paper presents an FPGA friendly implementation of a Gaussian Radial Basis SVM well suited to classification of grayscale images. We identify a novel optimization of the SVM formulation that dramatically reduces the computational inefficiency of the algorithm. The implementation achieves 88.6% detection accuracy in gender classification which is to the same degree of accuracy of software implementations using the same classification mechanism.

An integral part of interactive computing environments are systems that have the ability to process information about their users in real-time. In many cases it is desirable to not only recognize a human user but also to extract as much information about the user as possible, such as gender, ethnicity, age, etc. In this paper we present an FPGA implementation of a neural network configured specifically for performing face detection and gender classification in real-time video streams. Our streaming architecture performs the face and gender classification tasks at 30 frames per second on a small sized Virtex-4 FPGA, at accuracy comparable to that of a leading commercial software implementation.

Algorithm-architecture co-exploration is hindered by the lack of efficient tools. As a consequence, designers are currently able to explore only a limited set of points in the whole design space. Therefore, a tool that can allow fast exploration of algorithmic and architectural tradeoffs in an automated manner is highly desired. In this paper, we describe TANOR an automated tool targeted for designing hardware accelerators for the class of N-body interaction problems. The design flow, starting from a high level (MATLAB) description, configures the entire system automatically. We describe the design of TANOR and demonstrate the effectiveness and adaptability of our tool using three different target applications, namely, the gravitational kernel used in astrophysics, the gaussian kernel common in image processing applications, and a force calculation kernel applied in molecular dynamics. Our results demonstrate that TANOR generates hardware accelerator that are competitive with existing custom accelerator.

Discrete droplet digital microfluidics-based biochips face problems similar to that in other VLSI CAD systems, but with new constraints and interrelations. We focus on one such problem of resource constrained scheduling for digital microfluidic biochips. Since the problem is NP-complete, finding the optimal solution is a very time expensive task. We propose a hybrid priority scheduling algorithm solution directly applicable to digital microfluidics with the potential to yield near optimal schedules in the general case in a very short time. Furthermore we propose the use of configurable detectors that allow for even more improved system performance.