From self-driving cars to high dynamic range (HDR) imaging, the demand for image-based applications is growing quickly. In mobile systems, these applications place particular strain on performance and energy efficiency. As traditional memory systems are optimized for 1D memory access, they are unable to efficiently exploit the multi-dimensional locality characteristics of image-based applications which often operate on sub-regions of 2D and 3D image data. We have developed a new Patch Memory System (PMEM) tailored to application domains that process 2D and 3D data streams. PMEM supports efficient multidimensional addressing, automatic handling of image boundaries, and efficient caching and prefetching of image data. In addition to an optimized cache, PMEM includes hardware for offloading structured address calculations from processing units. We improve average energy-delay by 26% compared to EVA, a memory system for computer vision applications. Compared to a traditional cache, our results show that PMEM can reduce processor energy by 34% for a selection of CV and IP applications, leading to system performance improvement of up to 32% and energy-delay product improvement of 48-86% on the applications in this study.

In a typical portable multimedia system, external access, which is usually dominated by block-based video content, induces more than half of total system power. Embedded compression (EC) effectively reduces external access caused by video content by reducing the data size. In this paper, an algorithm and a hardware architecture of a new type EC codec engine with multiple modes are presented. Lossless mode, and lossy modes with rate control modes and quality control modes are all supported by single algorithm. The proposed four-tree pipelining scheme can reduce 83% latency and 67% buffer size between transform and entropy coding. The proposed EC codec engine can save 62%, 66%, and 77% external access at lossless mode, half-size mode, and quarter-size mode and can be used in various system power conditions. With TSMC 0.18 mum 1P6M CMOS logic process, the proposed EC codec engine can encode or decode CIF 30 frame per second video data and achieve power saving of more than 109 mW. The EC codec engine itself consumes only 2 mW power.

iVisual, an intelligent visual sensor SoC integrating 2790fps CMOS image sensor and 76.8GOPS, 374mW vision processor, is implemented on a 7.5mmx9.4mm die with 0.18μm CIS process. The light-in, answer-out SoC architecture avoids privacy problems of intelligent visual sensor. The feature processor and inter-processor synchronization scheme together increase 51% of average throughput. The 205GOPS/W power efficiency and 1.16GOPS/mm2 area efficiency are achieved.

Diastolic arrays are arrays of processing elements that communicate exclusively through First-In First-Out (FIFO) queues. FIFO virtualization units enable relaxed timing of data transfers, and include hardware support to guarantee bandwidth and buffer space for all data transfers, which may follow composite paths through the network. We show that the architecture of diastolic arrays enables efficient synthesis from high-level specifications of communicating finite state machines so average throughput is maximized. Preliminary results are presented on an H.264 decoding benchmark.

Since motion-compensated temporal filtering (MCTF) becomes an important temporal prediction scheme in video coding algorithms, this paper presents an efficient temporal prediction engine which not only is the first MCTF hardware work but also supports traditional motion-compensated prediction (MCP) scheme to provide computation scalability. For the prediction stage of MCTF and MCP schemes, modified extended double current Frames is adopted to reduce the system memory bandwidth, and a frame-interleaved macroblock pipelining scheme is proposed to eliminate the induced data buffer overhead. In addition, the proposed update stage architecture with pipelined scheduling and motion estimation (ME)-like motion compensation (MC) with level C+ scheme can also save about half external memory bandwidth and eliminate irregular memory access for MC. Moreover, 76.4% hardware area of the update stage is saved by reusing the hardware resources of the prediction stage. This MCTF chip can process CIF 30 fps in real-time, and the searching range is [-32, 32) for 5/3 MCTF with four-decomposition level and also support 1/3 MCTF, hierarchical B-frames, and MCP coding schemes in JSVM and H.264/AVC. The gate count is 352-K gates with 16.8 KBytes internal memory, and the maximum operating frequency is 60 MHz.

A 124 MSamples/s JPEG 2000 codec is implemented on a 20.1 mm2die with 0.18 mum CMOS technology dissipating 385 mW at 1.8 V and 42 MHz. This chip is capable of processing 1920times1080 HD video at 30 fps. For previous works, the tile-level pipeline scheduling is used between the discrete wavelet transform (DWT) and embedded block coding (EBC). For a tile with size 256times256, it costs 175 kB on-chip SRAM for the architectures using on-chip tile memory or costs 310 MB/s SDRAM bandwidth for the architectures using off-chip tile memory. In this design, a level-switched scheduling is developed to eliminate tile memory and the DWT and the EBC are pipelined at pixel-level. This scheduling eliminates 175 kB on-chip SRAM and 310 MB/s off-chip SDRAM bandwidth. The level-switched DWT (LS-DWT) and the code-block switched EBC (CS-EBC) are developed to enable this scheduling. The codec functions are realized on an unified hardware, and hardware sharing between encoder and decoder reduces silicon area by 40%

The on-chip line buffer dominates the total area and power of line-based 2-D discrete wavelet transform (DWT). In this paper, a memory-efficient VLSI implementation scheme for line-based 2-D DWT is proposed, which consists of two parts, the wordlength analysis methodology and the multiple-lifting scheme. The required wordlength of on-chip memory is determined firstly by use of the proposed wordlength analysis methodology, and a memory-efficient VLSI implementation scheme for line-based 2-D DWT, named multiple-lifting scheme, is then proposed. The proposed wordlength analysis methodology can guarantee to avoid overflow of coefficients, and the average difference between predicted and experimental quality level is only 0.1 dB in terms of PSNR. The proposed multiple-lifting scheme can reduce not only at least 50% on-chip memory bandwidth but also about 50% area of line buffer in 2-D DWT module.

The on-chip line buffer dominates the total area and power of line-based 2-D DWT. Therefore, the line buffer wordlength has to be carefully designed to maintain the quality level due to the dynamic range growing and the round-off errors. In this paper, a complete analysis methodology is proposed to derive the required wordlength of line buffer given the desired quality level of reconstructed image. The proposed methodology can guarantee to avoid overflow of coefficients, and the difference between predicted and experimental quality level is averagely 0.06 dB in terms of PSNR.

Motion-compensated temporal filtering (MCTF) is an innovative prediction scheme for video coding and the core technology of scalable extension of H.264/AVC. The first MCTF and ME hardware work is in this paper. The proposed hardware not only can support the various coding schemes in JSVM and H.264 but also can adapt itself to provide rate-distortion-computation scalability. With the frame-level searching range data reuse and frame-interleaved MB pipelining scheme, external memory bandwidth are reduced 33%, and 10 Kbits buffer are saved. The proposed MCTF/ME hardware is of 350K gate counts and 30 KB internal buffer, which can perform four-level MCTF or H.264 P-/B-frames at CIF format

Motion-compensated temporal filtering (MCTF) is an open-loop prediction scheme, so the frame-level data reuse for MCTF is possible. In this paper, we propose two general frame-level data reuse schemes which can minimize the memory bandwidth of current and reference frames, respectively. And their relationships between the required memory bandwidth and the number of searching range buffers are also formulated under the constraint of the data dependency in joint scalable video model. Finally, we extend our analysis to pyramid MCTF and the impact of the inter-layer prediction scheme is also considered.

Motion-compensated temporal filtering (MCTF) is the core technology of the next generation scalable video coding schemes. There are two key steps in MCTF process, prediction step and update step. In this paper, an efficient scheme along with the hardware architecture of update step for VLSI implementation is proposed. The proposed update step scheme can reduce 55% off-chip memory access and turn the irregular access into regular access. 25% on-chip memory access is also reduced in deriving inverse motion vectors. 80% hardware area is saved by reusing the hardware resources of prediction step.

In this paper, a precompression quality-control algorithm is proposed. It can greatly reduce computational power of the embedded block coding (EBC) and memory requirement to buffer bit streams. By using the propagation property and the randomness property of the EBC algorithm, rate and distortion of coding passes is approximately predicted. Thus, the truncation points are chosen before actual coding by the entropy coder. Therefore, the computational power, which is measured with the number of contexts to be processed, is greatly reduced since most of the computations are skipped. The memory requirement, which is measured with the amount required to buffer bit streams, is also reduced since the skipped contexts do not generate bit streams. Experimental results show that the proposed algorithm reduces the computational power of the EBC by 80% on average at 0.8 bpp compared with the conventional postcompression rate-distortion optimization algorithm. Moreover, the memory requirement is also reduced by 90%. The average PSNR degrades only about 0.1-0.3 dB, on average.

The memory issue is the most critical problem for a high performance JPEG2000 architecture. The tile memory occupies more than 50% of the area in conventional JPEG2000 architectures. To solve this problem, we propose a stripe pipeline scheme. For this scheme, a level switch discrete wavelet transform (LS-DWT) and a code-block switch embedded block coding (CS-EBC) are proposed. With small additional memory, the LS-DWT and the CS-EBC can process multiple levels and code-blocks in parallel by an interleaved scheme. As a result, the overall memory requirements of the proposed architecture can be reduced to only 8.5% compared with conventional architectures.

In wireless video streaming application, video information may be corrupted by a noisy channel. By introducing error resilience and error concealment techniques, many researchers have tried to eliminate quality degradation of reconstructed picture in decoding a corrupted data. On the contrary, there are relatively fewer works discussing the ways to diminish the corruption. Hence, system designers need to use different methods to restrict the error cause by channel within a tolerable extent. In other words, the system will be difficult to be implemented in practical design. In this paper, we propose a way to protect the video header information in application layer without modifying standardized syntax. Beside, we also consider channel condition of wireless transmission and propose a way to reduce redundant bits used in channel coding. By doing this, the bitstream can be simply transmitted over practical wireless network and the reconstructed picture quality outperforms the original one.