The strong temporal consistency of surveillance video enables compelling compression performance with traditional methods, but downstream vision applications operate on decoded image frames with a high data rate. Since it is not straightforward for applications to extract information on temporal redundancy from the compressed video representations, we propose a novel system which conveys temporal redundancy within a sparse decompressed representation. We leverage a video representation framework called ADΔER to transcode framed videos to sparse, asynchronous intensity samples. We introduce mechanisms for content adaptation, lossy compression, and asynchronous forms of classical vision algorithms. We evaluate our system on the VIRAT surveillance video dataset, and we show a median 43.7% speed improvement in FAST feature detection compared to OpenCV. We run the same algorithm as OpenCV, but only process pixels that receive new asynchronous events, rather than process every pixel in an image frame. Our work paves the way for upcoming neuromorphic sensors and is amenable to future applications with spiking neural networks.

Neuromorphic "event" cameras, designed to mimic the human vision system with asynchronous sensing, unlock a new realm of high-speed and high-dynamic-range applications. However, researchers often either revert to a framed representation of event data for applications, or build bespoke applications for a particular camera's event data type. To usher in the next era of video systems, accommodate new event camera designs, and explore the benefits of asynchronous video in classical applications, we argue that there is a need for an asynchronous, source-agnostic video representation. In this paper, we introduce a novel, asynchronous intensity representation for both framed and non-framed data sources. We show that our representation can increase intensity precision and greatly reduce the number of samples per pixel compared to grid-based representations. With framed sources, we demonstrate that by permitting a small amount of loss through the temporal averaging of stable pixel values, we can reduce our representational sample rate by more than half, while incurring a drop in VMAF quality score of only 4.5. We also demonstrate lower latency than the state-of-the-art method for fusing and transcoding framed and event camera data to an intensity representation, while maintaining 2000X the temporal resolution. We argue that our method provides the computational efficiency and temporal granularity necessary to build real-time intensity-based applications for event video.

This paper presents the implementation and evaluation of a 50,000-pose-sample-per-second, 6-degree-of-freedom optical head tracking instrument with motion-to-pose latency of 28μs and dynamic precision of 1-2 arcminutes. The instrument uses high-intensity infrared emitters and two duo-lateral photodiode-based optical sensors to triangulate pose. This instrument serves two purposes: it is the first step towards the requisite head tracking component in sub- 100μs motion-to-photon latency optical see-through augmented reality (OST AR) head-mounted display (HMD) systems; and it enables new avenues of research into human visual perception - including measuring the thresholds for perceptible real-virtual displacement during head rotation and other human research requiring high-sample-rate motion tracking. The instrument's tracking volume is limited to about 120×120×250 but allows for the full range of natural head rotation and is sufficient for research involving seated users. We discuss how the instrument's tracking volume is scalable in multiple ways and some of the trade-offs involved therein. Finally, we introduce a novel laser-pointer-based measurement technique for assessing the instrument's tracking latency and repeatability. We show that the instrument's motion-to-pose latency is 28μs and that it is repeatable within 1-2 arcminutes at mean rotational velocities (yaw) in excess of 500°/sec.

Modern motion camera sensors capture light in discrete frames, a residual artifact from the earliest days of video capture that imposes an unnecessary synchronizing constraint on all pixels (all pixels must start and stop measuring at the same time). While fixed time frames do offer some simplicity -- simpler sensor design, simpler encoding, and simpler transmission -- the rigidity of frames imposes significant costs in terms of dynamic range, precision and sensitivity, and we claim that with advances in hardware and software the time is ripe for a paradigm shift. We present the design of a novel sensor architecture capable of producing a frameless, asynchronous stream of high dynamic range pixel data. The key design insight is to use a high resolution timer to measure the period of time required to gather a per-pixel configurable amount of light rather than the amount of light gathered in a fixed period of time. A system model detailing how such a sensor can be integrated into existing multimedia systems and applications is proposed, exposing a number of interesting challenges with respect to representing and compressing asynchronous pixel data. Results of a preliminary investigation of a baseline scheme that repurposes existing frame synchronous video encoding methods (i.e., H.264) are presented.

For generalized augmented reality to be feasible, the augmenting elements must be visible in varied environments and under rapidly changing, high dynamic range lighting, from bright sunlight to deep shadows. We present a high dynamic range, optical see-through, augmented reality display that dynamically adjusts the brightness of the virtual imagery to match the current brightness of the real scene. Critical components include the spatial brightness sensor array and the positional brightness image intensity matcher. The color, scene-adaptive HDR display system is based on a high-rate (15 kHz) DMD projector using a high-speed RGB LED illuminator, each color with independent 16 bit intensity control for each binary DMD frame. The critical input to the intensity matching algorithm is the output of an array of high sensitivity light sensors. This paper discusses the implementation of the system and reports performance via still and video demonstrations under a variety of lighting conditions.

We describe an augmented reality, optical see-through display based on a DMD chip with an extremely fast (16 kHz) binary update rate. We combine the techniques of post-rendering 2-D offsets and just-in-time tracking updates with a novel modulation technique for turning binary pixels into perceived gray scale. These processing elements, implemented in an FPGA, are physically mounted along with the optical display elements in a head tracked rig through which users view synthetic imagery superimposed on their real environment. The combination of mechanical tracking at near-zero latency with reconfigurable display processing has given us a measured average of 80 µs of end-to-end latency (from head motion to change in photons from the display) and also a versatile test platform for extremely-low-latency display systems. We have used it to examine the trade-offs between image quality and cost (i.e. power and logical complexity) and have found that quality can be maintained with a fairly simple display modulation scheme.

This paper introduces the first exact method for optimal resource sharing in a pipelined system in order to minimize area. Given as input a dependence graph and a throughput requirement, our approach searches through the space of legal resource allocations, performing both scheduling and optimal buffer insertion, in order to produce the minimum area implementation. Furthermore, we do not arbitrarily limit the number of concurrent threads or data tokens; instead, we explore the full space of legal token counts, effectively allowing the depth of pipelining to be determined by our algorithm, while concurrently minimizing area and meeting performance constraints. Our approach has been automated, and compared with an existing single-token scheduling approach. Experiments using a set of benchmarks indicate that our multi-token approach has significant advantages: (i) it can find schedules that deliver higher throughput than the single-token approach; and (ii) for the same throughput, the multi-token approach obtains solutions that consumed 33-61% less area.

In this work, we explore the lossless compression of 32-bit floating-point buffers on graphics hardware. We first adapt a state-of-the-art 16-bit floating-point color and depth buffer compression scheme for operation on 32-bit data and propose two specific enhancements: dynamic bucket selection and a Fibonacci encoder. Next, we describe a unified codec for any type of floating-point buffer: color, depth, geometry, and GPGPU data. We also propose a method to further compress variable-precision data. Finally, we test our techniques on color, depth, and geometry buffers from existing applications. Using our enhancements to an existing technique, we have improved bandwidth savings by an average of 1.26x. Our unified codec achieved average bandwidth savings of 1.5x, 7.9x, and 2.9x for color (including buffers incompressible by past work), depth, and geometry buffers. Even higher savings were achieved when combined with our variable-precision technique, though specific ratios will depend on the tolerance of the application to reducing its precision.

No abstract available.

In this paper we explore the problem of scheduling and allocation for asynchronous systems under latency, area, energy, and power constraints, and present exact methods for minimizing an implementation for either latency, area, or energy. This approach utilizes the the branch-and-bound strategy developed in [1], but targets a much more robust solution space by incorporating many-to-many mappings of operations to function units and energy and power considerations into the search space. Unlike many recent solutions that adapt synchronous methods to the asynchronous realm, our approach specifically targets the asynchronous domain. As a result, our solver's complexity and performance are independent of the discretization of time. We illustrate the effectiveness of this approach by running 36 different test cases on small and large input specifications; results are produced in 60 seconds or less for each example.

We present an energy model for a graphics processing unit (GPU) that is based on the amount and type of work performed in various parts of the unit. By designing and running directed tests on a GPU, we measure the energy consumed when performing different arithmetic and memory operations, allowing us to accurately predict the energy that any arbitrary mix of operations will take. With some knowledge of how data travels through and is transformed by the graphics pipeline, we can predict how many of each operation will occur for a given scene, leading to an estimate of the energy usage. We validate our model against different types of existing graphical applications. With an average difference of 3% from measured energy under typical workloads, our model can be used for various purposes. In this work, we explore and present two use cases: 1) predicting the energy performance of applications on a different architecture, and 2) exploring the energy efficiency of different algorithms to achieve the same graphical effect.

No abstract available.

A high-throughput low-latency digitalfinite impulse response(FIR) filter has been designed for use inpartial-response maximum-likelihood(PRML) read channels of modern disk drives. The filter is a hybrid synchronous-asynchronous design. The speed-critical portion of the filter is designed as a high-performance asynchronous pipeline sandwiched between synchronous input and output portions, making it possible for the entire filter to be embedded within a clocked system. A novel feature of the filter is that the degree of pipelining is dynamically variable, depending upon the input data rate. This feature is critical in obtaining a very low filter latency throughout the range of operating frequencies. The filter is a ten-tap six-bit FIR filter, fabricated in a 0.18-μm CMOS process. Resulting chips were fully functional over a wide range of supply voltages, and exhibited throughputs of over 1.3 giga-items/s, and latencies of 2-5 clock cycles. Interestingly, the filter throughput was limited by the synchronous portion of the chip; the internal asynchronous pipeline was estimated to be capable of significantly higher throughputs, around 1.8 giga-items/s. More importantly though, the adaptively pipelined nature of the filter allows it to offer a worst-case latency of only 10 ns, which is half the worst-case latency of the best previously reported comparable fully-synchronous implementation by Rylov et al.

This paper presents a fast analytical method for estimating the throughput of pipelined asynchronous systems, and then applies that method to develop a fast solution to the problem of pipelining ldquoslack matchingrdquo. The approach targets systems with hierarchical topologies, which typically result when high-level (block structured) language specifications are compiled into data-driven circuit implementations. A significant contribution is that our approach is the first to efficiently handle architectures with choice (i.e. the presence of conditional computation constructs such if-then-else and conditional loops).

In mobile devices, limiting the Graphics Processing Unitpsilas (GPUpsilas) energy usage is of great importance to extending battery life. This paper focuses on the first stage of the graphics processor pipeline - the vertex transformation stage - and introduces an approach to lowering its switching activity by reducing the precision of arithmetic operations. As a result, the approach enables a tradeoff between energy efficiency and the quality of the rendered image. This paper makes the following specific contributions: 1) a transition-based energy model for quantifying energy consumed as a function of arithmetic precision, and 2) detailed simulation results on several real-world graphics applications to evaluate the tradeoff between energy and precision. In most examples, over 23% of the energy can be saved by lowering arithmetic precision while still maintaining a faithful reproduction of the full-precision image. Pushing the idea further, over 36% energy can be saved by further lowering the precision while preserving acceptable result accuracy. We assert that this represents a significant energy savings that warrants further investigation and extension of our approach to the remaining stages of the graphics processor pipeline.

An asynchronous pipeline style is introduced for high-speed applications, called MOUSETRAP. The pipeline uses standard transparent latches and static logic in its datapath, and small latch controllers consisting of only a single gate per pipeline stage. This simple structure is combined with an efficient and highly-concurrent event-driven protocol between adjacent stages. Post-layout SPICE simulations of a ten-stage pipeline with a 4-bit wide datapath indicate throughputs of 2.1-2.4 GHz in a 0.18-mum TSMC CMOS process. Similar results were obtained when the datapath width was extended to 16 bits. This performance is competitive even with that of wave pipelines, without the accompanying problems of complex timing and much design effort. Additionally, the new pipeline gracefully and robustly adapts to variable speed environments. The pipeline stages are extended to fork and join structures, to handle more complex system architectures.

A new class of asynchronous pipelines is proposed, called lookahead pipelines (LP), which use dynamic logic and are capable of delivering multi-gigahertz throughputs. Since they are asynchronous, these pipelines avoid problems related to high-speed clock distribution, such as clock power, management of clock skew, and inflexibility in handling varied environments. The designs are based on the well-known PSO style of Williams and Horowitz as a starting point, but achieve significant improvements through novel protocol optimizations: the pipeline communication is structured so that critical events can be detected and exploited earlier. A special focus of this work is to target extremely fine-grain or gate-level pipelines, where the datapath is sectioned into stages, each consisting of logic that is only a single level deep. Both dual-rail and single-rail pipeline implementations are proposed. All the implementations are characterized by low-cost control structures and the avoidance of explicit latches. Post-layout SPICE simulations, in 0.18-mum technology, indicate that the best dual-rail LP design has more than twice the throughput (1.04 giga data items/s) of Williams' PSO design, while the best single-rail LP design achieves even higher throughput (1.55 giga data items/s).

This paper introduces a high-throughput asynchronous pipeline style, called high-capacity (HC) pipelines, targeted to datapaths that use dynamic logic. This approach includes a novel highly-concurrent handshake protocol, with fewer synchronization points between neighboring pipeline stages than almost all existing asynchronous dynamic pipelining approaches. Furthermore, the dynamic pipelines provide 100% buffering capacity, without explicit latches, by means of separate pullup and pulldown control for each pipeline stage: neighboring stages can store distinct data items, unlike almost all existing latchless dynamic asynchronous pipelines. As a result, very high throughput is obtained. Fabricated first-input-first-output (FIFO) designs, in 0.18-m technology, were fully functional over a wide range of supply voltages (1.2 to over 2.5 V), exhibiting a corresponding range of throughputs from 1.0-2.4 giga items/s. In addition, an experimental finite-impulse response (FIR) filter chip was designed and fabricated with IBM Research, whose speed-critical core used an HC pipeline. The HC pipeline exhibited throughputs up to 1.8 giga items/s, and the overall filter achieved 1.32 giga items/s, thus obtaining 15% higher throughput and 50% lower latency than the fastest previously-reported synchronous FIR filter, also designed at IBM Research.

This paper introduces a novel approach to efficiently implement several useful architectural features in asynchronous application-specific ICs (ASICs). These features include speculation, preemption, and eager evaluation, which have so far only been available on CPUs, and have not been adequately investigated for custom ASICs.

We present a technique for increasing the throughput of stream processing architectures by removing the bottlenecks caused by loop structures. We implement loops as self-timed pipelined rings that can operate on multiple data sets concurrently. Our contribution includes a transformation algorithm which takes as input a high-level program and gives as output the structure of an optimized pipeline ring. Our technique handles nested loops and is further enhanced by loop unrolling. Simulations run on benchmark examples show a 1.3 to 4.9x speedup without unrolling and a 2.6 to 9.7x speedup with twofold loop unrolling.

This paper presents an architecture and a wrapper synthesis approach for the design of multi-clock systems-on-chips. We build upon the initial work on multi-clock latency-insensitive systems by Singh and Theobald (2004), and provide a detailed system architecture with the following capabilities and benefits: (i) modules arc stalled only when needed, thereby avoiding unnecessary stalling, (ii) adequate metastability resolution is provided, (iii) handshake interfaces between modules are high-performance and low-latency, i.e., capable of transferring data packets on every clock cycle, (iv) IP cores with large clock distribution delays are correctly handled, and (v) an automated approach is provided for wrapper synthesis from formal specifications. For wrapper synthesis, we have developed an automated tool which accepts interface specifications in a high-level language (Component Wrapper Language, or CWL), and automatically produces gate-level implementations of wrapper circuitry that will correctly and efficiently stall the synchronous modules depending on the availability of I/O channels. An optimization is introduced to reduce the cost of the wrapper circuitry by eliminating "busy waiting." A small set of benchmark examples is also proposed, and synthesis results for the tool are promising.

Recently, reconfigurable architectures, which outperform DSP processors, have become important. Although many compilers have been developed on a source-level, there are several practical benefits to translating the binary targeted to popular processors onto reconfigurable architectures. However, the translated code could be less optimized. In particular, it is difficult to optimize memory accesses on a binary to exploit pipeline parallelism. This paper introduces dynamic binary translation and memory access optimization to overcome the limitations of static binary translation for reconfigurable architecture. The experimental results show a promising speedup up to 3.02 compared with the code whose memory accesses is not optimized in the pipeline fashion.

We propose a methodology for testing ultra-high-speed asynchronous pipelines, the latest and most promising asynchronous circuit design style. Unlike traditional delay-insensitive asynchronous micro-pipelines, which use slow capture-pass latches, these circuits employ aggressive handshaking protocols and transparent latches between fine-grain pipeline stages, in order to achieve high performance. Their functional robustness, however, relies on certain timing constraints that need to be satisfied. As a result, these circuits are no longer delay-insensitive, which means that stuck-at faults are not always leading to pipeline stalling. In addition, delay faults may result in violation of these timing constraints, thus affecting not only performance, as in delay-insensitive micro-pipelines, but also functional correctness. To address these new challenges, we develop a test method for both stuck-at and timing constraint violation faults in fine-grain ultra-high-speed asynchronous pipelines. The efficiency of the proposed method is demonstrated on MOUSETRAP, a recently developed pipeline for high-speed applications

We introduce a technique to rapidly generate summed-area tables using graphics hardware. Summed area tables, originally introduced by Crow, provide a way to filter arbitrarily large rectangular regions of an image in a constant amount of time. Our algorithm for generating summed-area tables, similar to a technique used in scientific computing called recursive doubling, allows the generation of a summed-area table in O(log n) time. We also describe a technique to mitigate the precision requirements of summed-area tables. The ability to calculate and use summed-area tables at interactive rates enables numerous interesting rendering effects. We present several possible applications. First, the use of summed-area tables allows real-time rendering of interactive, glossy environmental reflections. Second, we present glossy planar reflections with varying blurriness dependent on a reflected object’s distance to the reflector. Third, we show a technique that uses a summed-area table to render glossy transparent objects. The final application demonstrates an interactive depth-of-field effect using summedarea tables. Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Three-Dimensional

Latency-insensitive systems were recently proposed by Carloni et al. as a correct-by-construction methodology for single-clock system-on-a-chip (SoC) design using predesigned IP blocks. Their approach overcomes the problem of long latencies of global interconnects in deep-submicron technologies, while still maintaining much of the inherent simplicity of synchronous design. In particular, wires whose latency is greater than a clock cycle are segmented using "relay stations", and IP blocks are made robust to arbitrary communication delays. This paper shows, however, that significant extensions are needed to make latency-insensitive systems useful for the practical design of large-scale SoC's. In particular, this paper proposes three extensions. The first extension allows each synchronous module to treat its input and output channels in a much more flexible manner, i.e., with greater decoupling. The second extension generalizes inter-module communication from point-to-point channels to more complex networks of arbitrary topologies. Finally, the third extension is to target multi-clock SoC's. The net impact of our extensions is the potential for improved throughput, reduced power consumption, and greater flexibility in design.

The recent explosion in the number of handheld multimedia devices has created a need for energy-efficient computation due to limited battery lifetimes. We focus on multiplication, which is needed in several application domains, e.g., 3D graphics, signal processing, and cryptography. We introduce an asynchronous implementation of a plain Booth multiplier (i.e., radix-2), which is both area- and energy-efficient, and therefore suitable for mobile applications. This paper makes the following contributions. First, a novel counterflow organization is introduced, in which the data bits flow in one direction, and the Booth commands piggyback on the acknowledgments flowing in the opposite direction. Second, the arithmetic and shifter units are merged together to obtain significant improvement in area, energy as well as speed. Third, our design performs overlapped execution of multiple iterations of the Booth algorithm. Finally, the design is quite modular, which allows scaling to arbitrary operand widths, without gate resizing or cycle time overheads. Spice simulations in a 0.18 /spl mu/m TSMC process at 1.8 V, indicate promising performance: the multiplier takes 1.08 ns per Booth iteration, regardless of the operand widths, thereby demonstrating the scalability of our approach. In addition, the multiplier is fully functional at reduced supply voltages (e.g., 1.0 V), and thus capable of dynamically trading off performance for energy efficiency.

Many approaches recently proposed for high-speed asynchronous pipelines are applicable only to linear datapaths. However, real systems typically have non-linearities in their datapaths, i.e. stages may have multiple inputs ('joins') or multiple outputs ('forks'). This paper presents several new pipeline templates that extend existing high-speed approaches for linear dynamic logic pipelines, by providing efficient control structures that can accommodate forks and joins. In addition, constructs for conditional computation are also introduced. Timing analysis and SPICE simulations show that the performance overhead of these extensions is fairly low (5% to 20%).

A new asynchronous pipeline design is introduced for high-speed applications. The pipeline uses simple transparent latches in its datapath, and small latch controllers consisting of only a single gate per pipeline stage. This simple stage structure is combined with an efficient transition-signaling protocol between stages. Initial pre-layout HSPICE simulations of a 10-stage FIFO on a 16-bit wide datapath indicate throughput of 3.51 GigaHertz in 0.25 /spl mu/ CMOS, using a conservative process. This performance is competitive even with that of wave pipelines, without the accompanying problems of complex timing and much design effort. Additionally, the new pipeline gracefully and robustly adapts to variable-speed environments. The stage implementations are extended to fork and join structures, to handle more complex system architectures.

Logical initializability is the property of a gate-level circuit whereby it can be driven to a unique start state when simulated by a three-valued (0, 1, X) simulator. In practice, commercial logic and fault simulators often require initialization under such a three-valued simulation model. In this paper, the first sound and systematic synthesis method is proposed to ensure the logical initializability of synchronous finite-state machines. The method includes both state assignment and combinational logic synthesis steps. It is shown that a previous approach to synthesis-for-initializability, which uses a constrained state assignment method, may produce uninitializable circuits. Here, a new state assignment method is proposed that is guaranteed correct. Furthermore, it is shown that combinational logic synthesis also has a direct impact on initializability; necessary and sufficient constraints on combinational logic synthesis are proposed to guarantee that the resulting gate-level circuits are logically initializable. The above two synthesis steps have been incorporated into a computer-aided design tool, SALSIFY, targeted to both two-level and multilevel implementations.

We present a new method for the synthesis-for initializability of asynchronous state machines. Unlike existing approaches, our method incorporates a state assignment step. State assignment is critical, since initializability may be precluded by poor choice of state assignment. Also, we present precise conditions on hazard-free logic, to insure that the circuits are physically initializable.