Lossy image compression is often limited by the simplicity of the chosen loss measure. Recent research suggests that generative adversarial networks have the ability to overcome this limitation and serve as a multi-modal loss, especially for textures. Together with learned image compression, these two techniques can be used to great effect when relaxing the commonly employed tight measures of distortion. However, convolutional neural network-based algorithms have a large computational footprint. Ideally, an existing conventional codec should stay in place, ensuring faster adoption and adherence to a balanced computational envelope.As a possible avenue to this goal, we propose and investigate how learned image coding can be used as a surrogate to optimise an image for encoding. A learned filter alters the image to optimise a different performance measure or a particular task. Extending this idea with a generative adversarial network, we show how entire textures are replaced by ones that are less costly to encode but preserve a sense of detail.Our approach can remodel a conventional codec to adjust for the MS-SSIM distortion with over 20% rate improvement without any decoding overhead. On task-aware image compression, we perform favourably against a similar but codec-specific approach.

Deep learning for image and video compression has demonstrated promising results both as a standalone technology and a hybrid combination with existing codecs. However, these systems still come with high computational costs. Deep learning models are typically applied directly in pixel space, making them expensive when resolutions become large.In this work, we propose an online-trained upsampler to augment an existing codec. The upsampler is a small neural network trained on an isolated group of frames. Its parameters are signalled to the decoder. This hybrid solution has a small scope of only 10s or 100s of frames and allows for a low complexity both on the encoding and the decoding side.Our algorithm works in offline and in zero-latency settings. Our evaluation employs the popular x265 codec on several high-resolution datasets ranging from Full HD to 8K. We demonstrate rate savings between 8.6% and 27.5% and provide ablation studies to show the impact of our design decisions. In comparison to similar works, our approach performs favourably.

CNN-based stereo matching methods achieve great performance but come with high computational requirements. Pruning a CNN can reduce the complexity but may in turn lead to memory conflicts, lowering throughput. Our proposed architecture and memory mapping technique aim at reducing conflicts to exploit extremely sparse stereo matching networks. To maintain a high utilization of processing elements, we decompose the de-convolution operation into several convolution operations. The proposed architecture provides a 2.1 x speed up over SCNN. Compared to the software implementation, only 0.01% performance drop is observed, so that the proposed architecture obtains state-of-the-art accuracy compared to existing sparsity aware hardware implementations.

Sending local data to cloud servers is vulnerable to user privacy, and its long update latency. Meanwhile, the state-of- the-art stereo matching method is still computation demanding, fine-tuning the whole model on-device is not a practicable solution because of the limited power budget and computation ability on edge devices. In this study, we propose a two-stage online stereo matching refinement system, using an additional light-weight network to learn the domain gap between local data and cloud training data. We define a load-gain ratio to evaluate computer efficiency. This refinement system has a much better load-gain ratio than fine-tune. (0.2 v.s. 35.7 operation overhead/accuracy gain) Nevertheless, we only disburse 0.2% of additional parameters and 0.7% additional computation as set by inference the stereo matching model. Thus, it would be a suitable choice for an online training scenario. With re-scheduling the training pipeline, we use a patch-based layer fusion technique and reduce the off-chip memory bandwidth by 97%.

In this study, we propose a constant memory hardware architecture that can support weighted mode, median, and joint bilateral filters, which is referred to as CMWMF. This work aims to meet the high memory and computation requirements of processing depth maps with a large number of depth candidates. In the proposed architecture, we leverage the geometry smoothing characteristic of natural images to reduce the static random access memory (SRAM) size for hardware implementation. The architecture preserves a constant number of disparity values instead of depending on the label count and size of the local supporting window. A novel weighted median search procedure is proposed, which assigns a computation to each input cycle, thereby rendering the process hardware friendly. An index-checking technique is proposed to process out-of-order joint histograms. We adopted the above-mentioned techniques in our architecture as they consume a constant SRAM size and supports multiple types of filters. As a result, this architecture reduces the SRAM size by 92.4% with a negligible decrease in performance. According to our analysis on the KITTI, and Middlebury datasets, and with actual depth cameras, the preserved information is sufficient. The proposed architecture is one of the most suitable depth refinement architectures for scenarios having a large number of depth candidates.

Belief propagation (BP)-based stereo matching has popular owing to its regularity and ability to yield promising results. Some commonly observed hardware-implementation challenges pertaining to the use of this algorithm are large memory requirements and trade-offs between speed and chip area, along with an increasing disparity range. The paper presents a hardware- and memory-efficient architecture for building a BP-based disparity estimation system capable of overcoming issues associated with large disparity ranges. The proposed architecture is memory-efficient owing to the regularity of its underlying algorithm. In addition, the improved hardware efficiency can be attributed to processing element modifications to demonstrate shareable characteristics. Results obtained in this study reveal a 67.8% reduction in required memory corresponding to a time-area term complexity of O(L(logL)2), where L denotes the disparity range. This result is in stark contrast to the O(L2logL) and O(L2) complexities observed in extant studies. Compared to state-of-the-art implementations, the proposed architecture offers an 86.2% gate count reduction for message update units at a disparity range of 512. These results confirm the proposed architecture's suitability for use in large disparity scenarios.

Digital media is ubiquitous and produced in ever-growing quantities. This necessitates a constant evolution of compression techniques, especially for video, in order to maintain efficient storage and transmission. In this work, we aim at exploiting non-local redundancies in video data that remain difficult to erase for conventional video codecs We design convolutional neural networks with a particular emphasis on low memory and computational footprint. The parameters of those networks are trained on the fly, at encoding time, to predict the residual signal from the decoded video signal. After the training process has converged, the parameters are compressed and signalled as part of the code of the underlying video codec. The method can be applied to any existing video codec to increase coding gains while its low computational footprint allows for an application under resource-constrained conditions. Building on top of High Efficiency Video Coding, we achieve coding gains similar to those of pretrained denoising CNNs while only requiring about 1% of their computational complexity Through extensive experiments, we provide insights into the effectiveness of our network design decisions. In addition, we demonstrate that our algorithm delivers stable performance under conditions met in practical video compression: our algorithm performs without significant performance loss on very long random access segments (up to 256 frames) and with moderate performance drops can even be applied to single frames in high-resolution low delay settings.

Scene Parsing is a crucial step to enable autonomous systems to understand and interact with their surroundings. Supervised deep learning methods have made great progress in solving scene parsing problems, however, come at the cost of laborious manual pixel-level annotation. Synthetic data as well as weak supervision have been investigated to alleviate this effort. Nonetheless, synthetically generated data still suffers from severe domain shift while weak labels often lack precision. Moreover, most existing works for weakly supervised scene parsing are limited to salient foreground objects. The aim of this work is hence twofold: Exploit synthetic data where feasible and integrate weak supervision where necessary. More concretely, we address this goal by utilizing depth as transfer domain because its synthetic-to-real discrepancy is much lower than for color. At the same time, we perform weak localization from easily obtainable image level labels and integrate both using a novel contour-based scheme. Our approach is implemented as a teacher-student learning framework to solve the transfer learning problem by generating a pseudo ground truth. Using only depth-based adaptation, this approach already outperforms previous transfer learning approaches on the popular indoor scene parsing SUN RGB-D dataset. Our proposed two-stage integration more than halves the gap towards fully supervised methods when compared to previous state-of-the-art in transfer learning.

This paper introduces architecture with high throughput, low on-chip memory, and efficient data access for Improved Dense Trajectories (iDT) as video representations for realtime action recognition. The iDT feature can capture longterm motion cues better than any existing deep feature, which makes it crucial in state-of-the-art action recognition systems. There are three major features in our architecture design, including a low bandwidth frame-wise feature extraction, low on-chip memory architecture for point tracking, and two-stage trajectory pruning architecture for low bandwidth. Using TSMC 40nm technology, our chip area is 3.1 mm2, and the size of on-chip memory is 40.8 kB. The chip can support videos in resolution of 320×240 with a throughput of 203 fps under 215 MHz, which is a 81.2 times speedup compared with CPU. Under the same operating frequency, it can also provide feature extraction for six windows of size 320 × 240 in higher resolution videos with a throughput of 34 fps.

The belief propagation (BP) technique is successful in image stereo matching problem. However, when we consider stereo matching for videos, directly applying the BP algorithm frame by frame results in unsatisfactory temporally inconsistent disparity maps. In this paper, we present the temporally consistent belief propagation for video stereo matching. We introduce a temporal term in traditional BP objective function and propose an adaptive weighting scheme to account for this temporal term. We show that the proposed algorithm performs favorably against previous methods in the stereo video datasets. Furthermore, the proposed method can solve problems induced by previous methods like error propagation from previously occluded regions.

CNN-based super-resolution methods achieve great performance but come with high bandwidth requirements. This paper proposes a bandwidth efficient CNN-based architecture for super resolution supporting up to Full HD images at 60 fps. The bandwidth efficiency is achieved by layer fusion with data reuse scheduling and dynamic quantization. To maintain a high utilization of processing elements, the deconvolution operation is decomposed into a several convolution operations. The proposed architecture provides a 94.8 % bandwidth reduction. Compared to the software implementation, less than 1 % performance drop is induced in this VLSI implementation. The enables the proposed architecture to obtain state-of-the-art accuracy compared to existing super-resolution hardware implementations.

In this paper, we propose a compact hardware architecture which can support HOG, HOF, MBHx and MBHy feature extraction in real-time. To achieve low-cost hardware, we first design a hardware-oriented algorithm. In addition to simplification of complicated numeric computation, we also modify the computation of the gradient at boundary to reduce on-chip buffers. The architecture is highly parallel in order to fully reuse input data and to achieve high throughput. Other features of our architecture include a four-bank memory composed of four SRAMs, a 2-cycle weighted binning, and a 2-port memory for histogram collection. The final specification can support HOG, HOF, MBHx, and MBHy feature extraction with a throughput of 65 fps in full-HD videos operating at 215 MHz, consuming only 57.6 kB on-chip memory and a gate count of 411k.

This paper introduces an efficient hardware accelerated feature extraction architecture with a high spec of 1920×1080 image resolution at 120 fps. We choose MoFREAK feature [1] to implement in our real-time action recognition system. MoFREAK is a local spatio-temporal feature, which combines the appearance and motion descriptor independently. We design a two phase architecture to balance the throughput difference between feature detection and feature description. Binary-mask image is adopted to detect feature point location efficiently. For feature description, to reduce high bandwidth requirement for spatial-temporal MoFREAK features, block-based keypoint technique is proposed to reduce bandwidth for grouped features. The synthesis result of our proposed architecture in TSMC 40nm technology works at 200 MHz with 1039K gate counts provides 1.2K block-based features per frame at 120 fps and 0.5K blockbased features at 240 fps. With binary-mask image, we reduce about 88% cycles and bandwidth of scanning image. With blockbased keypoint, we reduce about 81% of the salinet points and keypoints. The combination of binary-mask image and blockbased keypoint reduces about 81% of the feature extraction system bandwidth.

An accurate segment-based cost aggregation method is proposed in this paper. Segment-based methods can speed up the procedure according to the assumption that pixels belonging the same segment should share the same physical property. One of the most critical problems of those methods is the limited pixels within the corresponding segment. To enlarge supporting region of aggregation efficiently, we include similar adjacent segments with a fixed weight instead of a complex one. With introducing the simplified spatial weight for adjacent segments, the proposed method outperforms existing segment-based cost aggregate method[11], the Adaptive Support-weight[3], and the cross-based cost aggregation[10]. Comparing the other aspect of performance, this algorithm also speeds up the computation time by 43%.

Digital refocusing has a tradeoff between complexity and quality when using sparsely sampled light fields for low-storage applications. In this paper, we propose a fast physically correct refocusing algorithm to address this issue in a twofold way. First, view interpolation is adopted to provide photorealistic quality at infocus-defocus hybrid boundaries. Regarding its conventional high complexity, we devised a fast line-scan method specifically for refocusing, and its 1D kernel can be 30× faster than the benchmark View Synthesis Reference Software (VSRS)-1D-Fast. Second, we propose a block-based multi-rate processing flow for accelerating purely infocused or defocused regions, and a further 3- 34× speedup can be achieved for high-resolution images. All candidate blocks of variable sizes can interpolate different numbers of rendered views and perform refocusing in different subsampled layers. To avoid visible aliasing and block artifacts, we determine these parameters and the simulated aperture filter through a localized filter response analysis using defocus blur statistics. The final quadtree block partitions are then optimized in terms of computation time. Extensive experimental results are provided to show superior refocusing quality and fast computation speed. In particular, the run time is comparable with the conventional single-image blurring, which causes serious boundary artifacts.

In this paper, we propose an accelerated local feature extraction in a reuse scheme for action recognition. Most local features of the previous frame could be reused due to the high correlation between successive frames. Feature extraction is only needed to be applied partially in the current frame. The full-frame features of the current frame are combined by features extracted at different times. Experimental results show that the proposed reuse scheme can achieve comparable performance to full-frame feature extraction while computational effort is reduced significantly.

Digital refocusing for sparsely sampled light fields results in aliasing effect. For realistic quality, previous works performed anti-aliasing by applying time-consuming view interpolation for a heuristic number of novel views. In this paper, we study this problem by first performing a spectral analysis to give an analytical rule for the novel view number, which saves 34% of views compared to the intuitional choice. Then we propose a fast refocusing algorithm using a view interpolation method which is about 30x faster than VSRS-1D-Fast. Experimental results show the effectiveness of our approach in terms of speed and quality. For showing realistic refocusing, a light-field wafer-level-optics array is implemented, and its refocused images are compared to pictures captured by a real camera.

This paper introduces a memory efficient architecture for belief propagation based disparity estimation. To find the bottleneck of the memory, a lifetime analysis of the exchanged message is presented. The analysis leads to architecture which can take advantage of the data characteristics resulting in memory reduction, computing resource and memory access. In the experimental result, the proposed architecture gains 20% speed up in software simulation compared to non-optimized counterpart. In hardware implementation, more than 42% area is reduced by this architecture design without affecting the performance. With further design effort 61.8% area reduction is achieved.

This paper introduces a highly integrated system providing very accurate object detection with RGB-D sensor. To solve the problem that there are always insufficient training sets for object detection in real world, we present an online learning architecture to learn templates and to detect objects real-time. The proposed novel concept skips the training phase required in previous recognition works, and it comprises independent tracking and detection function, which collaborates with each other to make the detection more precise. We furthermore illustrate four strategies for online learning and compare the efficiency. With depth information, the experiment results perform remarkable in challenging scenarios.

It is impractical to apply a general spike sorting algorithm for every subject because of the individual characteristics of brain signal. Furthermore, extracting more neural activities for higher accuracy of spike sorting requires more input electrodes as well as large power consumption and chip area. Therefore, several practical constraints are considered in this work when implementing a programmable spike sorting hardware with large number of input channels. In this paper, we provide a 128-channel spike detection processor for spike sorting microsystem without compromise of the power efficiency. This chip consumes only 87.02uW and 9.7uW/mm2of power density, fabricated with 90nm low-leakage CMOS process.

In this paper, we propose a novel concept of region-of-unpredictable (ROU) to accelerate full-frame feature generation in video sequences. Due to the high correlation between successive frames, there are only few regions in which the features could not be estimated accurately from the previous frame called region-of-unpredictable (ROU). We develop a scheme combining partial feature extraction in ROU with feature prediction from the previous frame. The full-frame features of the current frame can then be obtained to minimize information loss. Experimental results show that the ROU determination algorithm supports 95.71% detection rate. The full-frame feature generation scheme using ROU determination saves 79.38% computational time compared with the full-frame feature extraction.

As the study of the wireless body area sensor network (BASN) keeps growing, corresponding applications such as bio-signal processing for personal healthcare are gaining more attention. To realize a miniature and multi-functional system for biomedical applications, the design of on-sensor mircoprocessor is considered to be a solution. However, how to balance the power consumption and system scalability remains a challenge. In this work, we try to involve the concept of multi-core computing systems into biomedical applications. A fully programmable system architecture based on the proposed two-way pipeline processing unit (two-way PPU) is introduced and evaluated. The proposed two-way PPU is a computing unit that adopts the general purpose processor (GPP) to provide high system programmability, and is easy to be cascaded for system extension. According to the implementation results, the proposed architecture can save up to 91% of energy compared with conventional multi-core architecture in a four-core biomedical processing microsystem.

2-D-to-3-D conversion is an important step for obtaining 3-D videos, as a variety of monocular depth cues have been explored to generate 3-D videos from 2-D videos. As in a human brain, a fusion of these monocular depth cues can regenerate 3-D data from 2-D data. By mimicking how our brains generate depth perception, we propose a reliability-based fusion of multiple depth cues for an automatic 2-D-to-3-D video conversion. A series of comparisons between the proposed framework and the previous methods is also presented. It shows that significant improvement is achieved in both subjective and objective experimental results. From the subjective viewpoint, the brain-inspired framework outperforms earlier conversion methods by preserving more reliable depth cues. Moreover, an enhancement of 0.70-3.14 dB and 0.0059-0.1517 in the perceptual quality of the videos is realized in terms of the objective-modified peak signal-to-noise ratio and disparity distortion model, respectively.

The five papers in this special section are devoted to the topic of new software and hardware programs and services used for error-tolerant multimedia applications.

In this paper, an accurate and robust positioning system based on street view recognition is introduced. Vision-based technique is employed for dynamically recognizing shop or building signs on the GPS map. Two mechanisms including view-angle invariant distance estimation and path refinement are proposed for robust and accurate position estimation. Through the combination of visual recognition technique and GPS scale data, the real user location can be accurately inferred. Experimental results demonstrate that the proposed system is reliable and feasible. Compared with 20m error of position estimation provided by the GPS, our system only has 0.97m error estimation.

Brain computer interface based on Electroencephalogram can be used to control the external devices through the motor imagery, and may be the next-generation user computer interface. However, this system requires a significant amount of data for the supervised algorithm training. The collection of training data is time-consuming, which may impede the usage in the daily life. In this paper, the trade-off between the training data size and algorithm accuracy is first analyzed. Then the reusing of the generalized pre-recorded training data set is explored to further improve this trade off. According to the simulation results, 63.8% training data collection time can first be saved with only 3% accuracy degradation.

An efficient feature matching architecture targets at real-time video stabilization is revealed in this paper. For some applications, such as vehicular application, real-time video stabilization is needed to provide instant stable video input. However, feature matching is usually the bottleneck to achieve high performance. High speed feature matching architecture is proposed to accelerate the performance of video stabilization. Locality sensitive hashing (LSH) helps us realize the feature matching procedure in hardware implementation. By applying the proposed dynamic table allocation and on-chip cache mechanism, this work achieves 422K queries/s and real-time feature matching with 90% in memory reduction and more than 50% in relieving the feature bus burden of the system.

A sparse signal can be reconstructed from a small amount of random and linear measurements by solving a system of underdetermined equations. In this paper, we study the reconstruction problem while the system undergoes dynamic modifications. Resolving this problem from scratch requires high computational efforts. Therefore, we propose an efficient homotopy-based reconstruction algorithm with warmstart, named WarmL1. WarmL1 quickly updates the previous solution to the desired one. Based on the concept of homotopy, WarmL1 breaks the reconstruction procedure into simple steps, and solves the problem iteratively. Four possible applications are presented and discussed to demonstrate the usage of WarmL1 for different warm-start situations. Experiments on these applications are performed. The results show that WarmL1 achieves 3.2× to 37.5× speeding up or up to 1/5100 l2-error at the same computational cost compared to related works.

Feature matching is an indispensable process for object recognition, which is an important issue for wearable devices with video analysis functionalities. To implement a low-power SoC for object recognition, the proposed visual vocabulary processor (VVP) is employed to accelerate the speed of feature matching. The VVP can transform hundreds of 128-D SIFT vectors into a 64-D histogram for object matching by using the binary-tree-based architecture, and 16 calculators for the computations of the Euclidean distances are designed for each of the two processors in each level. A total of 126 visual words can be saved in the six-level hierarchical memory, which instantly offers the data required for the matching process, and more than 5 times of bandwidth can be saved compared with the non-binary-tree-based architecture. As a part of the recognition SoC, the VVP is implemented with the 65-nm CMOS technology, and the experimental results show that the gate count and the average power consumption are 280 K and 5.6 mW, respectively.

This paper introduces a power estimation scheme and generated results of SoC (System-on-Chip) fabricated with different process nodes extending to very deep submicron tech nology. Different power modeling strategies are used to estimate power for analog and digital circuits. According to the analysis results, ultra low power analog components are key to successful biomedical SoC design if more advanced fabrication technology is utilized. Meanwhile, the digital part should be designed barely enough to serve the target application. Integrating more dedicated digital hardware accelerators can further reduce the total power consumption by lowering the working frequency of system processor. The goal of this paper is to provide a quantitative scheme to estimate the power consumption when SoC is fabricated with different process technologies. Then a suitable technology could be selected to manufacture the SoC for biomedical usage.

Accurate spike sorting is important for neuroscientific and neuroprosthetic applications. The sorting of spikes depends on the features extracted from the neural waveforms, and a better sorting performance usually comes with a higher sampling rate (SR). However for long duration experiments on free-moving subjects, the miniaturized and wireless neural recording ICs are the current trend. The compromise on sorting accuracy is usually made for the low power consumption with a lower SR. In this paper, the VLSI architecture of cubic spline interpolation is proposed to improve the power-accuracy tradeoff for the spike sorting microsystems. The window-based interpolation schedule, event-triggered processing, and two-step interpolation scheme are applied to save the memory and computation. 0.04 μW/channel is finally achieved after the implementation in 90 nm process.

Intensive home-care surveillance programs are associated with a marked decrease in the need for hospitalization. They can improve the functional statuses of elderly patients with severe congestive diseases. The GPU-based home-care surveillance system is effective and has a major impact on health expenditure than traditional surveillance equipments. In this work, we propose a spike sorting technique as a specific case for the GPU-based home surveillance system. Spike sorting is the procedure of classifying spikes corresponding to the firing neurons. In neuroscience research, spike sorting is adopted to analyze neural activities, brain functions and sensation. It is also a key component in cortically-controlled neuro-prosthetics for patients. In order to efficiently distinguish different neural spike activities, a robust spike sorting algorithm is required for above applications. To improve accuracy, multi-channel spike sorting is necessary. In addition, real-time monitoring for a home-care system is required. Therefore, we exploit a CUDA implementation using GPU for acceleration.

The paper exploits a vision-based intelligent vehicle cruise control system from the application level to the architecture level. Firstly, design considerations of the system are addressed in both computing power and accuracy aspects. Secondly, we present an efficient knowledge-based front-vehicle tracking algorithm. The algorithm yields below 5% error rate that outperforms the state-of-the-arts. Thirdly, a run-length-based algorithm optimization flow is introduced. Finally, specific hardware architecture is developed. It achieves 1280×960/80FPS and 4096×2160/10FPS requirements for multi-vehicle tracking tasks.

Multiview video (MVV) can provide viewers a complete 3D perception by its multiple-viewpoint feature. In order to support smooth viewpoint switching, virtual view synthesis is required. However the ultra high computational complexity and the system memory bandwidth requirement make the real-time multiview applications difficult. For the quad full-HD (QFHD) specification, 31.5GByte/Sec is needed for every virtual view. Furthermore, the conventional line buffer based or block based data scheduling cannot support the various geometries required in the virtual view synthesis. In the paper, a six-dimensional free-viewpoint view synthesis flow is proposed. By the proposed texture reorder and inverse reorder data interface, the on-chip data scheduling can be done under the conventional block pipelining even with different geometries. After integrating the proposed bandwidth reduction schemes, including the dynamic warping reference frame selection and on-chip texture buffer optimization, 95.7% system memory bandwidth is saved. Comparing with the previous 1D-horizontal-shifting-based 3DTV chips, the proposed scheduling provides full geometries including translations and rotations in the 3D space.

Owing to the diversity of projection surfaces, an effective mobile display system must be adaptive to the surface to avoid introducing a clipped scene. In this paper, we propose a smart mobile display system which automatically adapts to the location and motion of a surface. Firstly, an imperceptible structured light technique is adopted and continuous adaptation is accomplished. Secondly, a specifically designed code image with high distortion tolerance is proposed. Thirdly, we present a priority-based correction method to revise previous decoding results. Finally, a matching procedure resisting noise interruption is introduced. The system achieves 95% correct rate under the common indoor illuminance. In addition, the system performance is independent of the projected content and the surface shapes.

Building models by mimicking the structures and functions of visual cortex has always been a major approach to implement a human-like intelligent visual system. Several feed-forward hierarchical models have been proposed and perform well on invariant feature extraction. However, less attention has been given to the biologically plausible feature matching model which mimics higher levels of the ventral stream. In this work, with the inspirations from both neuroscience and computer science, we propose a framework for rapid object recognition and present the feature-selective hashing scheme to model the memory association in inferior temporal cortex. The experimental results on 1000-class ALOI dataset demonstrate its efficiency and scalability of learning on feature matching. We also discuss the biological plausibility of our framework and present a bio-plausible network mapping of the feature-selective hashing scheme.

The concept of perceptual integration of multiple depth cues is proposed for converting 2D videos to 3D videos. Several monocular depth cues have been explored to reconstruct 3D videos. We propose reliability-based promotion and fusion of depth cues for an automatic reconstruction of 3D videos in this demo by mimicking how human visual system does.

Loopy belief propagation (BP) is an effective solution for assigning labels to the nodes of a graphical model such as the Markov random field (MRF), but it requires high memory, bandwidth, and computational costs. Furthermore, the iterative, pixel-wise, and sequential operations of BP make it difficult to parallelize the computation. In this paper, we propose two techniques to address these issues. The first technique is a new message passing scheme named tile-based BP that reduces the memory and bandwidth to a fraction of the ordinary BP algorithms without performance degradation by splitting the MRF into many tiles and only storing the messages across the neighboring tiles. The tile-wise processing also enables data reuse and pipeline, resulting in efficient hardware implementation. The second technique is anO(L) fast message construction algorithm that exploits the properties of robust functions for parallelization. We apply these two techniques to a very large-scale integration circuit for stereo matching that generates high-resolution disparity maps in near real-time. We also implement the proposed schemes on graphics processing unit (GPU) which is four-time faster than standard BP on GPU.

This paper describes a method to stabilize video for vehicular applications based on feature analysis. An investigation on camera motion model is conducted. Harris features are extracted under the proposed resolution adaptation scheme. Besides, features are described with SURF-like descriptor. For feature matching, KD-tree with best-bin-first search significantly reduces the matching time. A damping filer is utilized to model and predict the unwanted oscillation. 93.1% correct rate in average is achieved in divergent driving conditions. Only 0.114 second is required to process a frame at resolution 1280×960. The provided benchmark shows outperformance of the proposed method.

Belief propagation (BP) is a popular global optimization technique in computer vision. However, it requires huge bandwidth and memory in hardware implementation because it iteratively processes messages between the neighboring nodes. In this paper, we propose an efficient message reduction algorithm that greatly reduces the bandwidth and memory consumption. Compared with the original message passing operation, we successfully reduce the memory and bandwidth with similar quality. For stereo matching of a VGA input where the disparity range is 64, the proposed algorithm can achieve 93.75% message memory reduction with about only 0.2-2.2% bad pixels quality degradation. The proposed algorithm greatly reduces the memory requirement and is suitable both for hardware and software realization.

This paper targets at detecting preceding vehicles in a wide range of distance. We propose an Adaboost-based approach combined with hierarchical image and sub-window scaling schemes. The relationship is investigated among object characteristics, image structures and image scales. A parameter set is developed to easily adjust overall performance, which benefits researchers to establish a vehicle detection system. It achieves 96.6% detection rate with 2.0% false alarm rate along proposed methodology. The benchmark of several learning-based vehicle detection approaches is also provided. The results show the outperformance of the proposed method.

This paper aims at detecting preceding vehicles in a variety of distance. A sub-region up-scaling scheme significantly raises far distance detection capability. Three frame pipeline structures involving object predictors are explored to further enhance accuracy and efficiency. It claims a 140-meter detecting distance along proposed methodology. 97.1% detection rate with 4.2% false alarm rate is achieved. At last, the benchmark of several learning-based vehicle detection approaches is provided.

Accurate spike sorting is an important issue for neuroscientific and neuroprosthetic applications. The sorting of spikes depends on the features extracted from the neural waveforms, and a better sorting performance usually comes with a higher sampling rate (SR). However the long duration experiment on free-moving subjects is the current trend. For low power considerations, the compromise on sorting accuracy is made for the miniaturized and wireless neural recording ICs with a lower SR. In this paper, we introduce the cubic spline interpolation to strike the power and accuracy tradeoff on the recording microsystems for the off-site and on-chip spike sorter. According to the simulation results, the recorder operated with the SR of 12.5k sample per second (sps) outperforms the system with 25ksps SR on both accuracy and power if the interpolation is appropriately performed before the spike sorting.

Neuromodulation of the brain is an emerging therapy to control the epileptic seizure. The therapy can be improved with a closed-loop mechanism in which the electrical stimuli is activated in accordance with the seizure onset. In this paper, a correlation integral (CI) processor in a form of application specific integrated circuit is designed to estimate the brain complexity, chaoticity, after the EEG/ECoG sensors. Since the neural firing becomes more organized prior to the seizure, the intent is to drive the neuromodulator after the early detection of the seizure onset. With the simplified CI algorithm and channel-folded architecture, 0.14μW/channel power consumption is achieved in 90nm CMOS process to simultaneously extract chaoticity for 16 channels in a real time. The simulation results demonstrate a 98.23% and 97.81% of sensitivity and specificity for the classification of normal and epileptic brain rhythms.

We propose a new architecture for stereo matching using belief propagation. The architecture combines our fast, fully-parallel processing element (PE) and memory-efficient tile-based BP (TBP) algorithm. On the architectural level, we develop several novel techniques, including a three stage pipeline, a message forwarding scheme, and a boundary message reuse scheme, which greatly reduce the required bandwidth and power consumption without sacrificing performance. The simulation shows that the architecture can generate HDTV720p results at 30 fps when operating at 227MHz. The high-quality depth maps enable real-time depth image based rendering and many other important applications in the 3D TV industry.

In the process of scalable video coding (SVC) decoding, large external memory bandwidth is required for SVC inter-layer prediction. In this paper, a low bandwidth decoder framework is proposed for SVC. Two main decoding schemes are developed to reduce the external memory bandwidth. Macroblock-level on-the-fly padding and on-line upsampling is proposed for SVC spatial scalability decoding. This scheme reduces 36% of decoding bandwidth and 34% of processing cycles. For SVC quality scalability, a layer-interleaving decoding scheme is proposed to eliminate all inter-layer prediction bandwidth which is 41-51% of decoding bandwidth. The corresponding hardware architectures of these two decoding schemes are also provided. This low bandwidth framework can save 33-52% of DRAM access power for SVC decoding.

As the history of television industry goes, multiview video (MW) and its applications draw more and more attentions by the realistic 3D scene it can bring. In these applications, virtual view synthesis is required for providing free view point sequences so as to fulfill a real-time display system. In this paper, a low-area architecture is proposed. By employing the linear-interpolated approximation algorithm, the large area requirement due to the synthesis parameters is resolved. In addition, redundant information for fraction bits of parameters is further reduced by precision fitting analysis. As a result, 95.9% of area for matrix parameter rendering stage and 69.5% for vector transform stage are reduced with only 0.0059 dB overhead of PSNR performance.

Multiview video (MVV) can provide viewers a complete 3D perception by its multiple-viewpoint feature. In order to support smooth viewpoint switching, virtual view synthesis is required. However, color and illumination information varies between views according to the light field conditions. This mismatch leads to ghost effects and wrong reflection phenomena. In this paper, a hybrid color compensation method is proposed. With the proposed inter-view color correspondence estimation, a smoothly changing color/illumination field is established between reference views. Furthermore, the reflection on mirrorlike materials can also be detected and optimized by the proposed hybrid reflection model. As a result, virtual view frames with better perceptual quality are generated. In the objective test, the proposed algorithm also provides 0.26-0.42 dB PSNR gains.

Image pipeline processing is crucial to generating high quality images in applications using complementary metaloxide-semiconductor (CMOS)/charge-coupled device sensors. The on-chip line buffer normally dominates the total area and power dissipation due to the needed filter window buffering. As image resolution and filter support increase, the area and power requirement increase accordingly. This paper presents a novel pyramid architecture to efficiently process a system that the image pipeline is between an image sensor and video coding engine. By utilizing the features of the pyramid structure and block-based video/image encoders, the proposed architecture is scalable from low to high image resolution and filter size. The input image is first partitioned into floors of tiles to reduce the frame line buffer. Two computing schemes, immediate result reuse and vertical snack scan, are utilized to reduce the overlapping redundant computations. A 90 nm CMOS chip design with 7 × 5 filter support for 3840 × 2160 quad full high definition video at 30 frames/s is designed to demonstrate the performance of power and area efficiency. Compared with traditional architectures with frame line buffers, the proposed design has shown that the power consumption is reduced by 25% to 108 mW from 145 mW. The chip area is reduced by 65% to 309 K from 888 K logic gates. The external memory bandwidth increases to 8286 Mbit/s from 5972 Mbit/s for YUV4:2:0, from 7963 Mbit/s for YUV4:2:2, and is reduced by 30% from 11944 Mbit/s for YUV4:4:4.

Belief propagation (BP) is an effective algorithm for solving energy minimization problems in computer vision. However, it requires enormous memory, bandwidth, and computation because messages are iteratively passed between nodes in the Markov random field (MRF). In this paper, we propose two methods to address this problem. The first method is a message passing scheme called tile-based belief propagation. The key idea of this method is that a message can be well approximated from other faraway ones. We split the MRF into many tiles and perform BP within each one. To preserve the global optimality, we store the outgoing boundary messages of a tile and use them when performing BP in the neighboring tiles. The tile-based BP only requires 1-5% memory and 0.2-1% bandwidth of the ordinary BP. The second method is an O(L) message construction algorithm for the robust functions commonly used for describing the smoothness terms in the energy function. We find that many variables in constructing a message are repetitive; thus these variables can be calculated once and reused many times. The proposed algorithms are suitable for parallel implementations. We design a low-power VLSI circuit for disparity estimation that can construct 440 M messages per second and generate high quality disparity maps in near real-time. We also implement the proposed algorithms on a GPU, which can calculate messages 4 times faster than the sequential O(L) method.

Belief propagation is a popular global optimization technique for many computer vision problems. However, it requires extensive computation due to the iterative message passing operations. In this paper, we present a new process element (PE) for efficient message construction. The efficiency is gained by exploiting the unique characteristics of the generalized Potts model (truncated linear mode) of the smoothness term in the Markov random field. For stereo estimation with L disparity values, the algorithm successfully reduces the computation from O(L2) to O(L) and retains the high throughput and low latency. Compared with the direct message construction PE, our method achieves 87.14% computation saving and a 94.38% PE area reduction.

For H.264/AVC decoder system, the motion compensation bandwidth comes from two parts, the reference data loading bandwidth and the equivalent bandwidth from DRAM access overhead latency. In this paper, a bandwidth-efficient cache-based MC architecture is proposed. It exploits both intra-MB and inter-MB data reuse and reduce up to 46% MC bandwidth compared to conventional scheme. To reduce the equivalent bandwidth from DRAM access overhead latency, the DRAM-friendly data mapping and access control scheme are proposed. They can reduce averagely 89.8% of equivalent DRAM access overhead bandwidth. The average MC burst length can be improved to 9.59 words/burst. The total bandwidth reduction can be up to 32∼71% compared to previous works.

To provide more vivid perception, more and more advanced features, like the 4k×2k resolution and the multiview functionality, are emerging for TV. For a multiview video coding (MVC) encoder, motion and disparity estimation (ME/DE) take at least half the hardware requirement. To solve these challenges, a cache-based integer ME/DE algorithm is proposed. With a cache memory as the search window buffer, a predictor-centered ME/DE algorithm is presented. The search range can be reduced to ±16 pixels with less than 0.1dB quality drop compared with full search algorithm. Based on this algorithm, an integer ME/DE chip design is realized. It can reduce 82% on-chip SRAM and 39% system bandwidth. Moreover, the search candidate requirement is also reduced by 79%. As the result, an ME/DE chip design for 4k×2k quad-HD H.264 and HDTV MVC is implemented.

Fractional motion estimation (FME) is widely used in video compression standards. In H.264/AVC, the precision of motion vector is down to quarter pixels to improve the coding efficiency. However, FME occupies over 45% of the computation complexity in an H.264 encoder and this high complexity limits the processing capability. In this paper, a single-iteration full search FME is proposed. By the algorithm and architecture co-optimization, the bandwidth to the frame buffer is reduced by 31%. Furthermore, 82% of circuit area for the Hadamard transformation and subtraction are saved from the direct implementation. Compared with prior arts, the proposed design supports 3.39 times higher throughput with only 0.02 dB PSNR drop. Thus, the specification of 4096 times 2160 quad full high definition H.264/AVC FME processing can be achieved.

Particle filtering is a popular algorithm for vision-based target tracking. Despite its effectiveness in many fields of tracking, however, the computation requirement of particle filters is high. In this paper we propose an algorithm and architecture for vision-based particle filters. The proposed algorithm can estimate objects' positions, sizes, and angles by using color histogram as the feature. We propose two hardware parallelization schemes - one makes use of particle-level parallel operation and the other utilizes particle-level plus pixel-level parallel operation. The experimental result shows the tracking accuracy of our test sequences is over 85%, and the proposed hardware architecture can run the particle filtering in real-time, with 31.35 frames per second in average.

Principal component analysis (PCA) spike sorting hardware in an integrated neural recording system is highly desired for wireless neuroprosthetic devices. However, a large memory is required to store thousands of spike events during the PCA training procedure, which impedes the on-chip implementation for the PCA training engine. In this paper, a mean pre-estimation method is proposed to save 99.01% memory requirement by breaking the algorithm dependency. According to the simulation result, 100 dB signal-to-error power ratio can be preserved for the resulting principal components. According to the implementation result, 6.07 mm2silicon area is required after a 283.16 mm2area saving for the proposed PCA training hardware.

An emerging class of neural prostheses aims to provide more aggressive performance by realizing advanced realtime signal processing algorithms in particular the spike sorting on chips. To support realtime spike sorting for 128 channels, the traditional fully parallel approach duplicating 128 processing units results in a large burden on chip area. The fully folding approach sharing one processor over 128 channels consumes large dynamic power in data caching. We propose to use the parallel-folding structure to optimally tradeoff the area and power. Our 128-channel spike sorting processor consumes 1.36 mm2area and 1.87 mW power in 90 nm process. 91.1% and 63.4% of the hardware resources (area×power) are reduced compared to the fully parallel and the fully folding approaches respectively.

Context-based adaptive binary arithmetic coding (CABAC) is a crucial part in H.264/AVC which provides a great compression gain. However, the throughput of CABAC decoder is limited due to the data dependency of decoding algorithm. This paper presents a branch selection hardware architecture which employs the two-symbol parallel decoder by providing all possible choices and then selecting the true one. The proposed CABAC decoder hardware architecture can decode 1.95~1.98 bins per cycle. It is implemented by UMC 90 nm technology with 82 k gate counts at 222 MHz. The maximum throughput is 410 Mbins/sec which is sufficient for decoding video sequence at Level 5.0.

Scalable Video Coding (SVC) is an advanced video compression technique that can support temporal, spatial, and quality scalability to terminals with different network conditions. SVC adopts layered coding techniques to improve coding efficiency for spatial and quality scalability. Upsampling and inter-layer prediction are two important mechanisms to remove redundant information between different layers. However, upsampling occupying around 75% memory bandwidth of SVC decoder results in serious performance degradation, especially for applications with high resolutions. Moreover, inter-layer prediction with complex scheduling leads to difficulties when mapping the SVC decoder in parallel. In this paper, we propose a method to parallelize the SVC decoder on a multi-core stream processor platform in both efficiency and flexibility. We focus on mapping issues of spatial scalability supporting with various resolutions of decoded frames. The experiment result proves the proposed design for SVC decoder reduces 95% memory bandwidth of the upsampling module in JSVM, performed on a single general-purpose processor.

Because video services are becoming popular on portable devices, power becomes the primary design issue for video coders nowadays. H.264/AVC is an emerging video coding standard which can provide outstanding coding performance and thus is suitable for mobile applications. In this paper, we target a power-efficient H.264/AVC encoder. The main power consumption in an H.264/AVC encoding system is induced by data access of motion estimation (ME). At first, we propose hardware-oriented algorithms and corresponding parallel architectures of integer ME (IME) and fractional ME (FME) to achieve memory access power reduction. Then, a parameterized encoding system and flexible system architecture are proposed to provide power scalability and hardware efficiency, respectively. Finally, our design is implemented under TSMC 0.18 mum CMOS technology with 12.84 mm2core area. The required hardware resources are 452.8 K logic gates and 16.95 KB SRAMs. The power consumption ranges from 67.2 to 43.5 mW under D1 (720 x 480) 30 frames/s video encoding, and more than 128 operating configurations are provided.

In a typical portable multimedia system, external access, which is usually dominated by block-based video content, induces more than half of total system power. Embedded compression (EC) effectively reduces external access caused by video content by reducing the data size. In this paper, an algorithm and a hardware architecture of a new type EC codec engine with multiple modes are presented. Lossless mode, and lossy modes with rate control modes and quality control modes are all supported by single algorithm. The proposed four-tree pipelining scheme can reduce 83% latency and 67% buffer size between transform and entropy coding. The proposed EC codec engine can save 62%, 66%, and 77% external access at lossless mode, half-size mode, and quarter-size mode and can be used in various system power conditions. With TSMC 0.18 mum 1P6M CMOS logic process, the proposed EC codec engine can encode or decode CIF 30 frame per second video data and achieve power saving of more than 109 mW. The EC codec engine itself consumes only 2 mW power.

In this paper we propose a novel approach to the detection of abandoned luggage in video surveillance. Candidates of abandoned luggage items which may pose potential security threats are first identified and localized by our proposed foreground-mask sampling technique. Our approach can deal with luggage pieces of arbitrary shape and color without the need for prior learning, and it works well under crowded and highly-cluttered situations. This localization of suspicious luggage items in the scene enables us to focus attention and subsequent processing solely on their neighborhoods. The owner of the luggage is then located and tracked to determine whether or not the luggage has been abandoned deliberately. A probability model using the MAP principle is employed to calculate a posteriori confidence score for the luggage-abandonment event, and an alarm will be automatically triggered if the certainty of luggage abandonment is higher than a pre-defined threshold. We show our results on the video datasets provided by the 2007 IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS 2007) and the 2006 IEEE International Workshop on Performance Evaluation of Tracking and Surveillance (PETS 2006).

iVisual, an intelligent visual sensor SoC integrating 2790fps CMOS image sensor and 76.8GOPS, 374mW vision processor, is implemented on a 7.5mmx9.4mm die with 0.18μm CIS process. The light-in, answer-out SoC architecture avoids privacy problems of intelligent visual sensor. The feature processor and inter-processor synchronization scheme together increase 51% of average throughput. The 205GOPS/W power efficiency and 1.16GOPS/mm2 area efficiency are achieved.

High Definition (HD) video compression enables vivid reproduction of scenes. However, Motion Estimation (ME) requires large memory capacity and huge memory bandwidth, which are undesirable in many platforms including ASIC and SoC. In this paper, an algorithm and architecture design of cache system and fast ME in HD H.264/AVC are proposed. With the proposed cache system and hardware-oriented fast ME algorithm, the rate-distortion performance is maintained within 0.03dB difference, the size of on-chip memory reduced to only 10% to 21% of original size, while the external memory bandwidth from cache refill is also 18% to 56% less than that of level C data reuse scheme with vertical ..64 search range.

3-D video will become one of the most important video technologies in the next generation of television. Due to ultra high data bandwidth requirement for 3-D video, effective compression technology becomes an essential part in the infrastructure. Thus stereo and multiview video coding (MVC) plays a critical role. However, MVC systems require much more computational complexity relative to mono-view video coding systems. Therefore, an ef..cient prediction scheme is necessary for encoding. In this paper, a new fast motion estimation (ME) algorithm is proposed. By utilizing disparity estimation (DE) to ..nd corresponding blocks between different views, the coding information such as motion vectors can be effectively shared and reused from the coded view channel. Therefore, the computation for ME in most view channels can be greatly reduced. Experimental results show that compared with the full search block matching algorithm applied to both ME and DE, the proposed algorithm saves 95% computation with near-FSBMA quality.

In this paper, a real-time augmented view synthesis system is proposed. With real-time consideration and augmented reality property, the proposed system provides a novel application for making car pillars transparent to enlarge the eyesight of the drivers. Thanks to the proposed trinocular depth estimation, online depth generation becomes possible through trinocular fast dense disparity estimation. With the proposed texture mapping free viewpoint depth image based rendering on the GPU, the processing speed for view interpolation achieves real-time. The computational power doesn’t cost much so that this real-time system is achievable on common computers. The experimental results show that the proposed system is able to project view synthesized video which is integrated with the background to make the user perceive seamless outside scene inside their car.

A JPEG XR chip for HD-Photo is implemented with 25 mm2area in TSMC 0.18 um CMOS 1P6M technology at 100 MHz. According to the simulation results, the 4:4:4 1920x1080 HD-Photo 20 frames/sec can be encoded smoothly.

External memory bandwidth is an important issue in System-on-Chip (SoC) systems. Especially in high definition (HD) video coding, the bandwidth requirement of off-chip memory is critical in video processing. In recent researches, embedded compression shows high potential on off-chip memory bandwidth reduction. Works about embedded compression have been done for low power applications. However, there is no suitable efficient embedded compression with good rate-distortion performance for high throughput applications. In this paper, an algorithm and hardware architecture of high performance lossy embedded compression is proposed to ease the bus congestion problem while keeping the latency low. Using the proposed algorithm, not only the high throughput requirement of HD video encoder is met, but also the hardware cost is relatively low. From our simulation, about 70% memory bandwidth is reduced with only 0.1dB PSNR degradation in 1080p HD video.

Video transmission over a heterogeneous network suffers a condition that it has required to satisfy many different constraints because of the variety limitations of client equipments and preferences of each user. In this condition, the video stream should have the flexibility feature. Scalable video coding is one of the solutions to serve this condition. In order to provide a better flexibility on scalability feature, more than one combination with video spatial size, temporal frame rate, and visual quality resolution are utilized. It is difficult to adapt the appropriate relation between the various combinations and user preference. In this paper, we propose an objectivity-derived algorithm with soft decision scheme for serving scalable video adaptation. This method can be used in existing equipments without much computation overhead. This useful and simple solution is not only a practical way in scalable video adaptation design, but also an efficient scheme of achieving subjective satisfaction.

Multiview video coding (MVC) is getting more and more attention in recent years because of the realistic perception it can bring. However, the current H.264-based coding scheme cannot get enough coding gain in the rate-distortion (R-D) performance. Furthermore, the view synthesis and random access between views are also design challenges. In this paper, a multiview hybrid coding system with texture-depth synthesis is proposed. By sharing the prediction information between texture and depth maps, the depth map can be reconstructed with reduced bitrate overhead. Then, after using matrix-based warping and adaptive disparity compensation, the R-D performance of the proposed system can be about 2 to 4 dB higher than the current MVC coding scheme at low bitrate range.

For an intelligent multi-camera multi-object surveillance system, object correspondence across time and space is important to many smart visual applications. In this paper, we propose a temporal and spatial consistent labeling algorithm for this demand. In the algorithm, an object corresponding database records the temporal and spatial consistency information for each segmented mask. With the database, the object-mask correlations are propagated through the propagation rules by analyzing mask splitting/merging conditions. In the spatial consistent labeling method, the homography warping and the earth mover’s distance are adopted to match same objects across different views. The earth mover’s distance solves the double matching problem, allows the algorithm to work normally under a small deviation of detected object locations, and makes pairing results have minimum global matching distances. The concept trusting-former-pairs-more is also adopted to avoid frequent pair switching if two objects are too close. The correct spatial labeling rate is about 89.25% in average. For online processing applications, the algorithm need not trace back to the past frames. The overall processing speed is about 10.24 frame per second (fps) with CIF size video running on a 2.8GHz general purpose CPU.

High Definition (HD) H.264/AVC video compression is the emerging necessity on nowadays home entertainment environment and so on. However, Although B-frame coding scheme provides better quality, only P-frame encoders are presented due to too high complexity and memory requirement. In this paper, a frame-parallel encoding scheme based on B-frame’s data independency is proposed. It can largely reduce the system memory bandwidth and improve the processing capability. Then, the proposed IME and FME scheduling can further enhance the hardware utilization for frame-parallel scheme. Finally, a case study is given to show that the proposed scheme can largely reduce 66% system bandwidth compared to direct implementation from previous P-frame encoder.

Data flow plays an important factor when design a multimedia multithreading streaming system. In this paper, a reconfigurable memory architecture is proposed to smooth the stream data flow to decrease maximum stream data bandwidth from the external memory. The reconfigurable memory not only can configure as data cache but also can configure as tightly couple memory depending on the application. In addition, heterogenous threads level configurability can adjust the optima on chip thread number in different pipeline stage. Experiment shows the proposed architecture can reduce at least 27% external bandwidth and increase at least 25% throughput in a graphic streaming example.

Since motion-compensated temporal filtering (MCTF) becomes an important temporal prediction scheme in video coding algorithms, this paper presents an efficient temporal prediction engine which not only is the first MCTF hardware work but also supports traditional motion-compensated prediction (MCP) scheme to provide computation scalability. For the prediction stage of MCTF and MCP schemes, modified extended double current Frames is adopted to reduce the system memory bandwidth, and a frame-interleaved macroblock pipelining scheme is proposed to eliminate the induced data buffer overhead. In addition, the proposed update stage architecture with pipelined scheduling and motion estimation (ME)-like motion compensation (MC) with level C+ scheme can also save about half external memory bandwidth and eliminate irregular memory access for MC. Moreover, 76.4% hardware area of the update stage is saved by reusing the hardware resources of the prediction stage. This MCTF chip can process CIF 30 fps in real-time, and the searching range is [-32, 32) for 5/3 MCTF with four-decomposition level and also support 1/3 MCTF, hierarchical B-frames, and MCP coding schemes in JSVM and H.264/AVC. The gate count is 352-K gates with 16.8 KBytes internal memory, and the maximum operating frequency is 60 MHz.

3-D video will become one of the most significant video technologies in the next-generation television. Due to the ultra high data bandwidth requirement for 3-D video, effective compression technology becomes an essential part in the infrastructure. Thus multiview video coding (MVC) plays a critical role. However, MVC systems require much more memory bandwidth and computational complexity relative to mono-view video coding systems. Therefore, an efficient prediction scheme is necessary for encoding. In this paper, a new fast prediction algorithm, content-aware prediction algorithm (CAPA) with inter-view mode decision, is proposed. By utilizing disparity estimation (DE) to find corresponding blocks between different views, the coding information, such as rate-distortion cost, coding modes, and motion vectors, can be effectively shared and reused from the coded view channel. Therefore, the computation for motion estimation (ME) in most view channels can be greatly reduced. Experimental results show that compared with the full search block matching algorithm (FSBMA) applied to both ME and DE, the proposed algorithm saves 98.4-99.1% computational complexity of ME in most view channels with negligible quality loss of only 0.03-0.06 dB in PSNR.

Low power motion estimation (ME) of H.264/AVC is an important research issue because of the growing mobile applications of H.264/AVC encoder. In this paper, low power cache algorithm and architecture for fast ME of H.264/AVC is proposed in order to replace the conventional search range (SR) memory. With the block translation (BT) cache architecture, search trajectory prediction (STP) prefetching algorithm, and ultra low power cache miss hiding (CMH) strategy, 35% SR memory writing power and 67% SR memory static power are reduced for D1 videos. Combining fast ME with the proposed cache provides the total solution for low power ME hardware.

This paper presented a novel depth map generation method - the short-term motion assisted color segmentation, which combines the pictorial, monocular and binocular depth cues of human vision. The proposed method utilizes a motion/edge registration technique to avoid the motion jitter error in common motion segmentation. And the motion/image segment adaptation algorithm matches the connected components with the motion segments. Even for static scene, the connected component algorithm is still working for the depth map generation. The experimental results show that the adaptation of motion and image segmentation improves quality and smoothness of the depth map both in the spatial and temporal domain.

In this demonstration, the core processing engines of two video applications, 3D video and intelligent video surveillance, are demonstrated. The developed algorithms and its VLSI design results are shown with hardware prototypes processing input video on-the-fly. In addition to the processing engine design, the development tools for efficiently designing these chips are also demonstrated.

Multiview video coding (MVC) systems require much more bandwidth and computational complexity relative to mono-view video systems. Thus, when designing a VLSI architecture for MVC systems, the hardware resource allocation is a critical issue. In this paper, we propose a new system bandwidth analysis scheme for various and complicated MVC structures. The precedence constraint in the graph theory is adopted for deriving the processing order of frames in a MVC system. In addition, current block centric scheduling (CBCS) and search window centric scheduling (SWCS) are proposed for MVC bandwidth analysis. By adopting data reuse schemes, several design points are explored with the aid of the proposed analysis scheme. The suitable hardware resource allocation can be easily determined.

Luminance and chrominance correction (LCC) is important in multi-view video coding (MVC) because it provides better rate-distortion performance when encoding video sequences captured by ill-calibrated multi-view cameras. This paper presents a robust and fast LCC algorithm based on motion compensated linear regression which reuses the motion information from the encoder. We adopt the linear weighted prediction model in H.264/AVC as our LCC model. In our experiments, the proposed LCC algorithm outperforms basic histogram matching method up to 0.4dB with only few computational overhead and zero external memory bandwidth. So, the dataflow of this method is suitable for low bandwidth/low power VLSI design for future multi-view applications.

Adaptive group-of-picture (GOP) structure is an important encoding tool in multi-level motion-compensated temporal filtering coding scheme. Compared to conventional fixed-GOP scheme, it can dynamically adapt the GOP size to enhance the coding performance based on each sequence's characteristics. But the existing adaptive GOP structure (AGS) algorithm proposed in JSVM requires huge computation complexity. In this paper, a fast AGS prediction algorithm is proposed. At first, based on the relationship among coding performance, GOP size and corresponding intra block ratio, a sub-GOP size prediction model for different decomposition levels is developed based on the encoded intra block ratio. Then, a prediction scheme is proposed to implement AGS by the sub-GOP size prediction model. It can predict the following sub-GOP size by current sub-GOP's information instead of searching all possible sub-GOP composition. The experimental results show that the proposed algorithm with linear threshold has almost equivalent coding performance as AGS in JSVM but only one-fourth computation complexity for 4-level interframe coding scheme is required.

This paper presented a novel dense disparity estimation method which is called as symmetric trinocular dense disparity estimation. Also a car surrounding camera array application is proposed to improve the driving safety by the proposed symmetric trinocular dense disparity estimation algorithm. The symmetric trinocular property is conducted to show the benefit of doing disparity estimation with three cameras. A 1D fast search algorithm is described to speed up the slowness of the original full search algorithms. And the 1D fast search algorithm utilizes the horizontal displacement property of the cameras to further check the correctness of the disparity vector. The experimental results show that the symmetric trinocular property improves the quality and smoothness of the disparity vector.

A 124 MSamples/s JPEG 2000 codec is implemented on a 20.1 mm2die with 0.18 mum CMOS technology dissipating 385 mW at 1.8 V and 42 MHz. This chip is capable of processing 1920times1080 HD video at 30 fps. For previous works, the tile-level pipeline scheduling is used between the discrete wavelet transform (DWT) and embedded block coding (EBC). For a tile with size 256times256, it costs 175 kB on-chip SRAM for the architectures using on-chip tile memory or costs 310 MB/s SDRAM bandwidth for the architectures using off-chip tile memory. In this design, a level-switched scheduling is developed to eliminate tile memory and the DWT and the EBC are pipelined at pixel-level. This scheduling eliminates 175 kB on-chip SRAM and 310 MB/s off-chip SDRAM bandwidth. The level-switched DWT (LS-DWT) and the code-block switched EBC (CS-EBC) are developed to enable this scheduling. The codec functions are realized on an unified hardware, and hardware sharing between encoder and decoder reduces silicon area by 40%

In an H.264/AVC video encoder, integer motion estimation (IME) requires 74.29% computational complexity and 77.49% memory access and becomes the most critical component for low-power applications. According to our analysis, an optimal low-power IME engine should be a parallel hardware architecture supporting fast algorithms and efficient data reuse (DR). In this paper, a hardware-oriented fast algorithm is proposed with the intra-/inter-candidate DR considerations. In addition, based on the systolic array and 2-D adder tree architecture, a ladder-shaped search window data arrangement and an advanced searching flow are proposed to efficiently support inter-candidate DR and reduce latency cycles. According to the implementation results, 97% computational complexity is saved by the proposed fast algorithm. In addition, 77.6% memory bandwidth is further saved with the proposed DR techniques at architecture level. In the ultra-low-power mode, the power consumption is 2.13 mW for real-time encoding CIF 30-fps videos at 13.5-MHz operating frequency

Due to the multiple reference frame motion estimation (MRF-ME), an H.264/AVC encoder requires ultrahigh memory bandwidth. Conventional multiple reference frames single current macroblock (MRSC) scheme only considers the data reuse within one frame, and requires on-chip memory size and off-chip memory bandwidth in proportional to the reference frame number. In this paper, a single reference frame multiple current macroblocks (SRMC) scheme is presented to further exploit the data reuse at frame level. With frame-level rescheduling of the motion estimation ME procedures in different reference frames, one loaded search window can be utilized by multiple current MBs in different original frames. The demanded on-chip memory size and off-chip memory bandwidth for MRF-ME can thus be reduced to those supporting only one reference frame. Moreover, based on SRMC scheme, an architecture prototype with two-stage mode decision flow is proposed. For HDTV specifications, 62.21 KB (74.8%) of SRAM and 364.3 MB/s (62.6%) of system bandwidth are saved in comparison with the MRSC scheme

The on-chip line buffer dominates the total area and power of line-based 2-D discrete wavelet transform (DWT). In this paper, a memory-efficient VLSI implementation scheme for line-based 2-D DWT is proposed, which consists of two parts, the wordlength analysis methodology and the multiple-lifting scheme. The required wordlength of on-chip memory is determined firstly by use of the proposed wordlength analysis methodology, and a memory-efficient VLSI implementation scheme for line-based 2-D DWT, named multiple-lifting scheme, is then proposed. The proposed wordlength analysis methodology can guarantee to avoid overflow of coefficients, and the average difference between predicted and experimental quality level is only 0.1 dB in terms of PSNR. The proposed multiple-lifting scheme can reduce not only at least 50% on-chip memory bandwidth but also about 50% area of line buffer in 2-D DWT module.

This paper presents a word-level decoding architecture of embedded block coding in JPEG 2000. This architecture decodes one coefficient per cycle based on the proposed word-level decoding algorithm. This algorithm eliminates state variable memories by decoding all bit-planes in parallel. The proposed column- switching scan order overcomes intra bit-plane dependency and inter bit-plane dependency to enable parallel processing. Implementation results show that the proposed architecture is capable of decoding 54 MSamples/s at 54 MHz, which can support HDTV 720p (1280 times 720, 4:2:2) decoding at 30 frames/s.

H.264/AVC is the latest video coding standard. It significantly outperforms the previous video coding standards, but the extraordinary huge computation complexity and memory access requirement make the hardwired codec solution a tough job. This paper describes the design methodology for H.264/AVC video codec. The system architecture and scheduling are addressed. The design consideration and optimization for its significant modules including bandwidth optimized motion compensation engine, reconfigurable intra predictor generator, low bandwidth parallel integer motion estimation are mentioned. Due to the complex, sequential, and highly data-depended characteristics of all essential algorithms in H.264/AVC, not only the pipeline structure but also efficient memory hierarchy is required. The design case with a hybrid task pipelining scheme, a balanced schedule with block-level, MB-level, and frame-level pipelining, are presented. By combining with many bandwidth reduction techniques and data reused schemes, very efficient architecture and implementation for plate-form based system is proved by the prototype chips.

The rapid progress of semiconductor technology enables the rising and flourishing of digital multimedia. Fancy terms such as ubiquitous multimedia, mobile multimedia, embedded video, etc., and hot products such as iPod, Camera phone, Portable Multimedia Player, etc., mark our colorful life, now and future. In this talk, I attempt to review the past and present developments of video coding, and to explore some future trends. Then, this talk will discuss the challenging optimization tasks that video codec architects have to face under the stringent time-to-market pressure for applications ranged from low-power mobile multimedia to high definition video. Design issues, optimization techniques, and case studies are presented. What we have learned is that multimedia SOC design is a beautiful but challenging dance among video codec HW/SW designers, transistors and video bits.

This paper presents a word-level decoding architecture of Embedded Block Coding (EBC) in JPEG 2000. This architecture decodes one coefficient per cycle based on the proposed word-level decoding algorithm. This algorithm eliminates state variable memories by decoding all bit-planes in parallel. The proposed column-switching scan order overcomes intra bit-plane dependency and inter bit-plane dependency to enable parallel processing. Implementation results show the proposed architecture can decode 54 MSamples/s at 54 MHz, which can support HDTV 720p (1280 × 720, 4:2:2) decoding at 30 frames/sec in real time.

In motion estimation, fast algorithms usually lead to an irregular searching flow, and the power reduction on architecture level is constrained for poor data reuse (DR). In this paper, a parallel IME hardware for H. 264/AVC is proposed to well combine the techniques on algorithm and architecture levels. The "2-D SAD Tree" is adopted to support intra-and inter-candidate DR for the content-adaptive parallel-VBS four step search algorithm. A ladder-shaped reference data arrangement is proposed to support DR in both horizontal and vertical directions, while an advanced searching flow is applied to reduce the latency cycles. After these two techniques, 77.6% power of search window SRAMs can be reduced. According to the implementation result, in ultra low power mode, only 1.424 mW is required for realtime encoding CIF 30fps videos with 13.5 MHz operation frequency.

The on-chip line buffer dominates the total area and power of line-based 2-D DWT. Therefore, the line buffer wordlength has to be carefully designed to maintain the quality level due to the dynamic range growing and the round-off errors. In this paper, a complete analysis methodology is proposed to derive the required wordlength of line buffer given the desired quality level of reconstructed image. The proposed methodology can guarantee to avoid overflow of coefficients, and the difference between predicted and experimental quality level is averagely 0.06 dB in terms of PSNR.

This paper presents a relative depth layer extraction system for monoscopic video, using multi-line filters and a layer selection algorithm. Main ideas are to extract multiple linear trajectory signals from videos and to determine their relative depths using the concept of motion parallax. The proposed superficial line model used for detecting slow moving objects provides sufficient taps within few frames to reduce frame buffer, while the closest-hit line model used for detecting fast motion objects provides few enough taps to prevent blurring. To increase the correctness of layer map, three-level layer map co-decision is used to compensate low texture region defect

In this paper, a power-scalable H.264 encoding system is provided with the efforts on both the algorithm and the architecture levels. For a start, a motion estimation (ME) pre-skip algorithm is adopted as a system-level power-scalable algorithm. In order to realize a dedicated hardware, a novel reconfigurable macro-block (MB) pipelining architecture is proposed. It can improve not only system flexibility but also hardware efficiency. Besides, it is also beneficial for power management with module-level gated clock insertion. According to simulation results, the proposed H.264 encoder can support power-scalable functionality in the range of about 20 to 90 mW with graceful quality degradation

Motion-compensated temporal filtering (MCTF) is an innovative prediction scheme for video coding and the core technology of scalable extension of H.264/AVC. The first MCTF and ME hardware work is in this paper. The proposed hardware not only can support the various coding schemes in JSVM and H.264 but also can adapt itself to provide rate-distortion-computation scalability. With the frame-level searching range data reuse and frame-interleaved MB pipelining scheme, external memory bandwidth are reduced 33%, and 10 Kbits buffer are saved. The proposed MCTF/ME hardware is of 350K gate counts and 30 KB internal buffer, which can perform four-level MCTF or H.264 P-/B-frames at CIF format

3D TV will become a prominent technology in the next generation. In this paper, a depth image based rendering system is proposed from algorithm level to hardware architecture level. We propose a novel depth image based rendering algorithm with edge-dependent Gaussian filter and interpolation to improve the rendered stereo image quality. Based on our proposed algorithm, a fully-pipelined depth image based rendering hardware accelerator is proposed to support real-time rendering. The proposed hardware accelerator is optimized in three steps. First, we analyze the effect of fixed point operation and choose the optimal wordlength to keep the stereo image quality. Second, a three-parallel edge-dependent Gaussian filter architecture is proposed to solve the critical problem of memory bandwidth. Finally, we optimize the hardware cost by the proposed hardware architecture. Only 1/21 amounts of vertical PEs and 1/11 amounts of horizontal PEs is needed by the proposed folded edge-dependent Gaussian filter architecture. Furthermore, by the proposed check mode, the whole Z-buffer can be eliminated during 3D image warping. In additions, the on-chip SRAMs can be reduced to 66.7 percent compared with direct implementation by global and local disparity separation scheme. A prototype chip can achieve real-time requirement under the operating frequency of 80 MHz for 25 SDTV frames per second (fps) in left and right channel simultaneously. The simulation result also shows the hardware cost is quite small compared with the conventional rendering architecture

Low power hardware design for entropy coding of H.264/AVC baseline profile encoder is urgent for the increasing mobile applications. However, previous works are poor in the power performance. In this paper, the first low power context-based adaptive variable length coding (CAVLC) scheme named the side information aided (SIA) symbol look ahead (SLA) one-pass CAVLC is proposed, with the non-zero and abs-one SIA flags. A reconfigurable architecture for the SLA module is also proposed to support the low power CAVLC scheme efficiently. The resultant hardware power is reduced by 69% to only 3.7 mW at 27 MHz and 1.8 V for CIF-sized video coding. The total logic gate count is 27 K gates

Motion-compensated temporal filtering (MCTF) is an open-loop prediction scheme, so the frame-level data reuse for MCTF is possible. In this paper, we propose two general frame-level data reuse schemes which can minimize the memory bandwidth of current and reference frames, respectively. And their relationships between the required memory bandwidth and the number of searching range buffers are also formulated under the constraint of the data dependency in joint scalable video model. Finally, we extend our analysis to pyramid MCTF and the impact of the inter-layer prediction scheme is also considered.

In this paper, the scalable architecture of the embedded block coding (EBC) in JPEG 2000, the bit-plane parallel EBC, is proposed. We provided the analysis and an unified design methodology for the bit-plane parallel EBC architecture. To design the bit-plane parallel EBC, there exists three critical difficulties. To overcome the difficulties, three algorithms are proposed. By use of the proposed algorithms, the external bandwidth of the EBC is reduced by 55% averagely, and the throughput of a 4 bit-planes parallel EBC is higher than a word-level EBC with 10 bit-plans parallel by 1.5 times at the bitrate of 1 bits per pixel.

In this paper, the low power design techniques from algorithm to architecture levels are proposed for fractional motion estimation in H.264/AVC. The proposed AMPD algorithm can reduce 50.8% power with up to 0.1 dB quality drop. The proposed parallel architecture with efficient memory hierarchy can efficiently reuse data and save 61.6% power. Furthermore, the power aware functionality is included. Our design can gracefully vary the quality degradation of 0.1-3.9 dB in response to the 22.58-1.64 mW power consumption. This power-oriented design is very efficient for different mobile applications in various power situations.

The first SRAM-based multi-symbol arithmetic encoder was proposed in this paper. Since several SRAM problems arise in this highly data-dependent operation, four methods were introduced to make it feasible. Based on data-forwarding architecture, modular banks with throw-backward/catch-forward and read/write isolation greatly enhanced the throughput. Our SRAM-based approach was implemented with 29%-35% of area compared to register-based design. Moreover, different throughput required in various applications could be attained by changing the number of SRAM banks. The proposed SRAM-based multi-symbol arithmetic encoder achieved high throughput and low cost at the same time

Motion-compensated temporal filtering (MCTF) is the core technology of the next generation scalable video coding schemes. There are two key steps in MCTF process, prediction step and update step. In this paper, an efficient scheme along with the hardware architecture of update step for VLSI implementation is proposed. The proposed update step scheme can reduce 55% off-chip memory access and turn the irregular access into regular access. 25% on-chip memory access is also reduced in deriving inverse motion vectors. 80% hardware area is saved by reusing the hardware resources of prediction step.

Weighted prediction (WP) is a tool to compensate the brightness difference in video sequences with brightness variations. In this paper, some weight parameter determination methods are surveyed, and a weighted prediction algorithm together with the hardware architecture design is proposed. The main idea of the algorithm is to limit the number of weight parameters transmitted by quantizing the parameter into levels and using only offset as the parameter. As a result, the extra parameters sent in each slice header is thus limited by the number of levels, and the parameter determination process requires much less computations. By further utilizing sub-sampling of brightness levels and estimated offset sum, a simplified architecture is also proposed. Simulation result shows that the later architecture achieves a coding gain of about 0.5 dB over weighted prediction method in JM9.6 and has a minimum overhead to hardware implementation.

Depth buffer bandwidth minimization with depth filter plays an important role in designing low power graphics processors. In this paper, an efficient adaptive tile depth filter (ATDF) algorithm is proposed. The key concept is to consider more occlusion conditions in order to achieve better performance. Two existing algorithms, Z/sub max/ and Z/sub max/ algorithms, are integrated to detect both occluded and non-occluded fragments. Moreover, two new techniques, coverage mask and adaptive tile mode, are proposed to further improve the performance. Experiments show that the proposed algorithm can filter out up to 85% fragments to reduce 40% memory bandwidth for test applications with complex scenes, which outperforms the previous works.

H.264/AVC significantly outperforms previous video coding standards with many new coding tools. However, the better performance comes at the price of the extraordinarily huge computational complexity and memory access requirement, which makes it difficult to design a hardwired encoder for real-time applications. In addition, due to the complex, sequential, and highly data-dependent characteristics of the essential algorithms in H.264/AVC, both the pipelining and the parallel processing techniques are constrained to be employed. The hardware utilization and throughput are also decreased because of the block/MB/frame-level reconstruction loops. In this paper, we describe our techniques to design the H.264/AVC video encoder for HDTV applications. On the system design level, in consideration of the characteristics of the key components and the reconstruction loops, the four-stage macroblock pipelined system architecture is first proposed with an efficient scheduling and memory hierarchy. On the module design level, the design considerations of the significant modules are addressed followed by the hardware architectures, including low-bandwidth integer motion estimation, parallel fractional motion estimation, reconfigurable intrapredictor generator, dual-buffer block-pipelined entropy coder, and deblocking filter. With these techniques, the prototype chip of the efficient H.264/AVC encoder is implemented with 922.8 K logic gates and 34.72-KB SRAM at 108-MHz operation frequency.

The memory bandwidth reduction for motion estimation is important because of the power consumption and limited memory bandwidth in video coding systems. In this paper, we propose a Level C+ scheme which can fully reuse the overlapped searching region in the horizontal direction and partially reuse the overlapped searching region in the vertical direction to save more memory bandwidth compared to the Level C scheme. However, direct implementation of the Level C+ scheme may conflict with some important coding tools and then induces a lower hardware efficiency of video coding systems. Therefore, we propose n-stitched zigzag scan for the Level C+ scheme and discuss two types of 2-stitched zigzag scan for MPEG-4 and H.264 as examples. They can reduce memory bandwidth and solve the conflictions. When the specification is HDTV 720p, where the searching range is [-128,128), the required memory bandwidth is only 54%, and the increase of on-chip memory size is only 12% compared to those of traditional Level C data reuse scheme.

3-D video will be the most prominent video technology in the next generation. Among the 3-D video technologies, stereo video systems are considered to be realized first in the near future. Stereo video systems require double bandwidth and more than twice the computational complexity relative to mono-video systems. Thus, an efficient coding scheme is necessary for transmitting stereo video. In this paper, a new structure of prediction core in stereo video coding systems is proposed from the algorithm level to the hardware architecture level. The joint prediction algorithm (JPA), which combines three prediction schemes, is proposed for high coding efficiency and low computational complexity. It makes the system outperform MPEG-4 temporal scalability and simple profile by 2-3 dB in rate-distortion performance. Besides, JPA also utilizes the characteristics of stereo video and successfully reduces about 80% computational complexity. Then, a new hardware architecture of the prediction core based on JPA and a modified hierarchical search block-matching algorithm is proposed. With a special data flow, no bubble cycles exist during the block-matching process. The proposed architecture also adopts the near-overlapped candidates reuse scheme to save the heavy burden of data access. Besides, both on-chip memory requirement and off-chip memory bandwidth can be reduced by the proposed new scheduling. Compared with the hardware requirement for the implementation of full search block-matching algorithm, only 11.5% on-chip SRAM and 3.3% processing elements are needed with a tiny PSNR drop, making it area-efficient while maintaining high stereo video quality and processing capability

In the new video coding standard H.264/AVC, motion estimation (ME) is allowed to search multiple reference frames. Therefore, the required computation is highly increased, and it is in proportion to the number of searched reference frames. However, the reduction in prediction residues is mostly dependent on the nature of sequences, not on the number of searched frames. Sometimes the prediction residues can be greatly reduced, but frequently a lot of computation is wasted without achieving any better coding performance. In this paper, we propose a context-based adaptive method to speed up the multiple reference frames ME. Statistical analysis is first applied to the available information for each macroblock (MB) after intra-prediction and inter-prediction from the previous frame. Context-based adaptive criteria are then derived to determine whether it is necessary to search more reference frames. The reference frame selection criteria are related to selected MB modes, inter-prediction residues, intra-prediction residues, motion vectors of subpartitioned blocks, and quantization parameters. Many available standard video sequences are tested as examples. The simulation results show that the proposed algorithm can maintain competitively the same video quality as exhaustive search of multiple reference frames. Meanwhile, 76 %-96 % of computation for searching unnecessary reference frames can be avoided. Moreover, our fast reference frame selection is orthogonal to conventional fast block matching algorithms, and they can be easily combined to achieve further efficient implementations.

In this paper, a precompression quality-control algorithm is proposed. It can greatly reduce computational power of the embedded block coding (EBC) and memory requirement to buffer bit streams. By using the propagation property and the randomness property of the EBC algorithm, rate and distortion of coding passes is approximately predicted. Thus, the truncation points are chosen before actual coding by the entropy coder. Therefore, the computational power, which is measured with the number of contexts to be processed, is greatly reduced since most of the computations are skipped. The memory requirement, which is measured with the amount required to buffer bit streams, is also reduced since the skipped contexts do not generate bit streams. Experimental results show that the proposed algorithm reduces the computational power of the EBC by 80% on average at 0.8 bpp compared with the conventional postcompression rate-distortion optimization algorithm. Moreover, the memory requirement is also reduced by 90%. The average PSNR degrades only about 0.1-0.3 dB, on average.

A computation-aware motion estimation algorithm is proposed in this paper. Its goal is to find the best block-matching results in a computation-limited and computation-variant environment. Our algorithm is characterized by a one-pass flow with adaptive search strategy. In the prior scheme, Tsai et al. propose that all macroblocks are processed simultaneously, and more computation is allocated to the macroblock with the largest distortion among the entire frame in a step-by-step fashion. This implies that random access of macroblocks is required, and the related information of neighboring macroblocks cannot be used to be prediction. The random access flow requires a huge memory size for all macroblocks to store the up-to-date minimum distortions, best motion vectors, and searching steps. On the contrary, our one-pass flow processes the macroblocks one by one, which can not only significantly reduce the memory size but also effectively utilize the context information of neighboring macroblocks to achieve faster speed and better quality. Moreover, in order to improve the video quality when the computation resource is still sufficient, the search pattern is allowed to adaptively change from diamond search to three step search, and then to full search. Last but not least, traditional block matching speed-up methods are also combined to provide much better computation-distortion curves

An 81 MSamples/s JPEG 2000 single-chip encoder is implemented on 5.5 mm/sup 2/ area using 0.25-/spl mu/m CMOS technology. This IC can losslessly encode HDTV 720p resolution at 30 frames/s in real time. Three techniques are adopted: line-based discrete wavelet transform, parallel embedded block coding, and precompression rate-distortion optimization. The line-based discrete wavelet transform achieves the minimum external memory access, while the internal memory is reduced by a proper memory access scheme. The parallel embedded block coding increases the throughput and reduces the memory bandwidth with similar hardware cost comparing to conventional architectures. By accurately estimating bit rates, the precompression rate-distortion optimization reduces the required computational power and processing time of the embedded block coding since the code-blocks are truncated before compression. Experimental results show that this encoder has the highest throughput with the smallest area compared with other designs in the literature.

The paper summarizes our design experiences of various image and video codec IPs. The design issues and methodology of custom video codecs are discussed. The design methodology can be summarized as four stages: system analysis; algorithm optimization; architecture exploration; code development. Based on these guidelines, several design cases are presented, including the proposed JPEG, MPEG-4, and H.264 architectures.

A four field variable block size motion compensated adaptive de-interlacing method is proposed to improve the accuracy of the motion vectors and lower the occlusions of motion compensated de-interlacing. The proposed de-interlacing method consists of variable block size motion estimation/compensation with four field SAD, interlaced block mode decision, and new block modes. The variable block size motion estimation and compensation improve the accuracy of the motion vectors, especially for spatially-periodic patterns. The new block modes and the interlaced block mode decision make block decisions more precisely, and special patterns that motion compensation cannot be compensated are correctly de-interlaced by these two methods. The subjective view shows an improvement of the accuracy of the motion vectors and the correctness of the mode decision.

The memory issue is the most critical problem for a high performance JPEG2000 architecture. The tile memory occupies more than 50% of the area in conventional JPEG2000 architectures. To solve this problem, we propose a stripe pipeline scheme. For this scheme, a level switch discrete wavelet transform (LS-DWT) and a code-block switch embedded block coding (CS-EBC) are proposed. With small additional memory, the LS-DWT and the CS-EBC can process multiple levels and code-blocks in parallel by an interleaved scheme. As a result, the overall memory requirements of the proposed architecture can be reduced to only 8.5% compared with conventional architectures.

To the best of authors' knowledge, this paper presents the first work on memory analysis of VLSI architectures for motion-compensated temporal filtering (MCTF). The open-loop MCTF prediction scheme has led the revolution for hybrid video coding methods that are mainly based on the close-loop MC prediction (MCP) scheme, and it also becomes the core technology of the coming video coding standard, MPEG-21 part 13-scalable video coding (SVC). In this paper, the macroblock (MB)-level and frame-level data reuse schemes are analyzed for the MCTF. The MB-level data reuse is especially for the motion estimation (ME), and the level C+ scheme is proposed, which can further reduce the memory bandwidth of the conventional level C scheme. Frame-level data reuse schemes for MCTF are proposed according to the open-loop prediction nature.

The motion-compensated temporal filtering (MCTF) is an innovative prediction scheme for video coding and has become the core technology of the coming video coding standard, MPEG-21 part 13 - scalable video coding (SVC). This paper provides the system analysis of MCTF for VLSI implementation, which includes computational complexity, external memory access, external storage size, and coding delay. The one-level MCTF is analyzed first, and a modified double current frames scheme is introduced to address the external memory access penalty that results from fractional-pel motion compensation (MC). Then the analysis is extended to multi-level MCTF, in which many important system issues will be explored. Finally, a real-life test case was given to compare the system requirements of many different MCTF schemes and the prediction scheme of H.264/AVC.

An efficient depth image based rendering with edge dependent depth filter and interpolation is proposed. The proposed method can solve the hole-filling problem in DIBR system efficiently with high quality. The PSNR of the proposed method is better than the previous work by 6 dB and the subjective view shows the quality is better. In addition to that, the number of instruction cycles is 3.7 percent compared with the previous work

This paper proposes a practical content-dependent low power DCT design with tolerable quality drop. Low power issue has become more and more important, especially for portable devices. Unfortunately, low-power design always brings in significant quality drop at the same time. This work not only achieves ultra low power dissipation but also remains tolerable quality drop (about 0.1dB) in most cases. The proposed architecture is based on distributed arithmetic architecture [1] and combined with more reliable PPA classification algorithm [2]. It can accurately control bit-level calculation and avoid unnecessary calculation to save power. This characteristic is powerful and very useful in video encoder systems, since coefficients after DCT and Q become zero with high probability. This part of power can be saved without causing undesired quality drop.

In wireless video streaming application, video information may be corrupted by a noisy channel. By introducing error resilience and error concealment techniques, many researchers have tried to eliminate quality degradation of reconstructed picture in decoding a corrupted data. On the contrary, there are relatively fewer works discussing the ways to diminish the corruption. Hence, system designers need to use different methods to restrict the error cause by channel within a tolerable extent. In other words, the system will be difficult to be implemented in practical design. In this paper, we propose a way to protect the video header information in application layer without modifying standardized syntax. Beside, we also consider channel condition of wireless transmission and propose a way to reduce redundant bits used in channel coding. By doing this, the bitstream can be simply transmitted over practical wireless network and the reconstructed picture quality outperforms the original one.

The College of Electrical Engineering and Computer Science at the National Taiwan University has identified the area of content science for media-rich life, broadly construed, as one of core areas for the college's future directions. One major aspect of this project is to develop the enabling technology for reconfigurable platforms for multimedia applications. Specifically, the goal is to develop reconfigurable platforms of system-on-a-chip (SoC) components for the applications to support the needs of multi-modal multimedia contents and to provide rapid system prototyping. The faculties in the College of Electrical Engineering and Computer Science have formed a multi-discipline team to develop such technology. Our team includes seven faculties and more than thirty students from the college. This short report describes the reconfigurable platform for content science research activities currently underway by our team. Our current activities include to develop the technology to analyze the critical path for avoiding hardware contention, to minimize the use of logic components, to design the multimedia IPs, to optimally route the bus and place the logic units, to design energy efficient cache, to evaluate the performance and power consumption, to design the algorithm for temporal floor-planning/placement.

This paper provides a survey of state-of-the-art hardware architectures for image and video coding. Fundamental design issues are discussed with particular emphasis on efficient dedicated implementation. Hardware architectures for MPEG-4 video coding and JPEG 2000 still image coding are reviewed as design examples, and special approaches exploited to improve efficiency are identified. Further perspectives are also presented to address the challenges of hardware architecture design for advanced image and video coding in the future.

This letter derives fast decomposition for the quadrature mirror filterbanks (QMFs) of the low power spectral band replication (SBR) tools in the MPEG high efficiency advanced audio coding (HE AAC) decoder. In contrast with the standard method where computation-intensive matrix operations are employed in the QMF, the proposed method decomposes the matrix operations into conventional discrete cosine transform of type II and III (DCT-II and DCT-III) and simple permutations for easy implementation. The computational complexity can be also reduced effectively by using fast algorithms for DCT.

A de-interlacing algorithm using adaptive 4-field global/local motion compensated approach is presented. It consists of block-based directional edge interpolation, same-parity 4-field motion detection, global/local motion estimation and compensation. The edges are sharper when the directional edge interpolation is adopted. The same parity 4-field motion detection and the 4-field local motion estimation detect the static areas and fast motion by four reference fields, and the global motion estimation detects the camera panning and zooming motions. The global and local motion compensation recover the interlaced videos to the progressive ones. Experimental results show that the peak signal-to-noise ratio of our proposed algorithm is 2/spl sim/3 dB higher than that of previous studies and attain the best quality of subjective view.

Mathematical morphology operations are applied in many real-time applications, such as video segmentation. For real-time requirement, efficient hardware implementation is necessary. This paper proposes a new architecture named Partial-Result-Reuse (PRR) architecture for mathematical morphological operations with flat structuring elements. Partial results generated during calculation process are kept and reused in this architecture to reduce hardware cost. With PRR concept and self-affinity property of structuring elements, the proposed architecture is more cost-effective and more general than existing morphology architectures. Moreover, it can be combined with systolic array to give consideration to both flexibility and hardware cost. We also propose a methodology to generate PRR architecture. With graphic method, the PRR architecture can be easily generated, and it can deal with structuring elements of any shape. The very large scale integration implementation of the PRR architecture shows that the area of processing element is small and can be fully piplelined without large overhead. The maximum frequency of the chip is 200 MHz in simulation, while processing speed of 550 morphological operations/s on a 720/spl times/480 frame can be achieved.

This paper presents a parallel architecture for the Embedded Block Coding (EBC) in JPEG 2000. The architecture is based on the proposed word-level EBC algorithm. By processing all the bit planes in parallel, the state variable memories for the context formation (CF) can be completely eliminated. The length of the FIFO (first-in first-out) between the CF and the arithmetic encoder (AE) is optimized by a reconfigurable FIFO architecture. To reduce the hardware cost of the parallel architecture, we proposed a folded AE architecture. The parallel EBC architecture can losslessly process 54 MSamples/s at 81 MHz, which can support HDTV 720p resolution at 30 frames/s.

Intra prediction with rate-distortion constrained mode decision is the most important technology in H.264/AVC intra frame coder, which is competitive with the latest image coding standard JPEG2000, in terms of both coding performance and computational complexity. The predictor generation engine for intra prediction and the transform engine for mode decision are critical because the operations require a lot of memory access and occupy 80% of the computation time of the entire intra compression process. A low cost general purpose processor cannot process these operations in real time. In this paper, we proposed two solutions for platform-based design of H.264/AVC intra frame coder. One solution is a software implementation targeted at low-end applications. Context-based decimation of unlikely candidates, subsampling of matching operations, bit-width truncation to reduce the computations, and interleaved full-search/partial-search strategy to stop the error propagation and to maintain the image quality, are proposed and combined as our fast algorithm. Experimental results show that our method can reduce 60% of the computation used for intra prediction and mode decision while keeping the peak signal-to-noise ratio degradation less than 0.3 dB. The other solution is a hardware accelerator targeted at high-end applications. After comprehensive analysis of instructions and exploration of parallelism, we proposed our system architecture with four-parallel intra prediction and mode decision to enhance the processing capability. Hadamard-based mode decision is modified as discrete cosine transform-based version to reduce 40% of memory access. Two-stage macroblock pipelining is also proposed to double the processing speed and hardware utilization. The other features of our design are reconfigurable predictor generator supporting all of the 13 intra prediction modes, parallel multitransform and inverse transform engine, and CAVLC bitstream engine. A prototype chip is fabricated with TSMC 0.25-/spl mu/m CMOS 1P5M technology. Simulation results show that our implementation can process 16 mega-pixels (4096/spl times/4096) within 1 s, or namely 720/spl times/480 4:2:0 30 Hz video in real time, at the operating frequency of 54 MHz. The transistor count is 429 K, and the core size is only 1.855/spl times/1.885 mm/sup 2/.

In this paper, three generic RAM-based architectures are proposed to efficiently construct the corresponding two-dimensional architectures by use of the line-based method for any given hardware architecture of one-dimensional (1-D) wavelet filters, including conventional convolution-based and lifting-based architectures. An exhaustive analysis of two-dimensional architectures for discrete wavelet transform in the system view is also given. The first proposed architecture is for 1-level decomposition, which is presented by introducing the categories of internal line buffers, the strategy of optimizing the line buffer size, and the method of integrating any 1-D wavelet filter. The other two proposed architectures are for multi-level decomposition. One applies the recursive pyramid algorithm directly to the proposed 1-level architecture, and the other one combines the two previously proposed architectures to increase the hardware utilization. According to the comparison results, the proposed architecture outperforms previous architectures in the aspects of line buffer size, hardware cost, hardware utilization, and flexibility.

A four field local motion compensated de-interlacing method is proposed to solve the occlusion and true motion vector problems of motion compensated de-interlacing. The proposed de-interlacing method consists of four field motion estimation, 5+5 taps ELA, and intra/MC macroblock mode decision. True motion vectors can be obtained by the four field motion estimation and its matching criterion is extended by the characteristic of interlace-scanned fields. The mode decision could detect an occlusive macroblock and choose its best mode. Hardware implementation has been considered. The results show that the proposed de-interlacing method can produce images compensated by a great majority of the original fields with little intra-field interpolation, and the subjective view results are jag-free and flicker-free high quality images.

We contributed a new VLSI architecture for fractional motion estimation of the H.264/AVC video compression standard. Seven inter-related loops extracted from the complex procedure are analyzed and two decomposing techniques are proposed to parallelize the algorithm for hardware with a regular schedule and full utilization. The proposed architecture, also characterized by a reusable feature, can support situations in different specifications, multiple standards, fast algorithms and some cost considerations. H.264/AVC baseline profile level 3 with complete Lagrangian mode decision can be realized with 290K gates at operating frequency of 100 MHz. It is a useful intellectual property (IP) design for platform based multimedia systems.

The large amount of the frame memory access and the die area occupied by the embedded internal buffer are the most critical issues for the implementation of the two-dimensional discrete wavelet transform (2D DWT). The former may consume the most power and waste the system memory bandwidth. The latter may enlarge the chip size and also consume much power. We categorize and analyze the 2D DWT architectures by different external memory scan methods. Then the overlapped stripe-based scan method is proposed to provide an efficient and flexible implementation for 2D DWT. The implementation issues of the internal buffer are also discussed, including the lifting-based and convolution-based. Some real-life experiments are given to show that the performance of area and power for the internal buffer is highly related to memory technology and working frequency, instead of the required memory bits only.

The critical path of the hardware for the global elimination algorithm (GEA) is too long to meet the real-time constraints for high-end applications. In this paper, we propose a new parallel GEA and its corresponding architecture. By dividing candidate blocks into independent groups and finding the most probable candidates of each group in parallel, instead of sequentially searching within the whole search range, parallel design can be developed as an array of GEA processing elements with much shorter critical path. Besides, the GEA processing element is optimized to reduce 30% of the gates, and the 2D data reuse is organized to save 80% of the SRAM bandwidth, which also reduces a lot of power. Simulation results show that our implementation can achieve real time processing of D1 30 Hz video with search range as H[-64, +63.5] V[-32, +31.5] while the operating frequency is 70 MHz, and the gate count is 113 K. Compared with full search, our gate count is six times smaller under the same frequency, and the PSNR loss is at most 0.1-0.2 dB.

In this paper, a novel pre-compression rate-distortion optimization algorithm is proposed, which can reduce computation power and memory requirement of JPEG 2000 encoder. It can reduce the wasted computational power of the entropy coder (EBCOT Tier-1) and unnecessary memory requirement for the code-stream. Distortion and rate of coding passes are calculated and estimated before coding, and therefore truncation point is selected before coding. Experimental results show that the computation time of EBCOT Tier-1 and memory requirement for the code-stream can be greatly reduced,especially at high compression ratio. The quality of the proposed algorithm is slightly lower than that of post-compression rate-distortion optimization algorithm.

This paper presents a new block matching motion estimation algorithm and its VLSI architecture design. The proposed global elimination algorithm (GEA) was derived from successive elimination algorithm (SEA), which can skip unnecessary sum of absolute difference (SAD) calculation by comparing minimum SAD with subsampled SAD (SSAD). Our basic idea is to separate the decision of early termination and SAD calculation for each candidate block to make data flow more regular and suitable for hardware. In short, we first compare the rough characteristics of all candidate blocks with the current block (SSAD). In turn, we select several best roughly matched candidate blocks to re-compare them with the current block by using detailed characteristics (SAD). Other features of GEA include fixed processing cycles, no initial guess, and high video quality (almost the same as full search). Unlike other fast algorithms, the mapping of GEA to hardware is very simple. We proposed an architecture that is composed of a systolic part to efficiently compute SSAD, an adder tree to support both SSAD and SAD calculations, and a comparator tree to avoid expensive sorting circuits. Simulation results show that our design is much more area efficient than many full-search architectures while maintaining high video quality and processing capability.

Automatic video segmentation plays an important role in real-time MPEG-4 encoding systems. Several video segmentation algorithms have been proposed; however, most of them are not suitable for real-time applications because of high computation load and many parameters needed to be set in advance. This paper presents a fast video segmentation algorithm for MPEG-4 camera systems. With change detection and background registration techniques, this algorithm can give satisfying segmentation results with low computation load. The processing speed of 40 QCIF frames per second can be achieved on a personal computer with an 800 MHz Pentium-III processor. Besides, it has shadow cancellation mode, which can deal with light changing effect and shadow effect. A fast global motion compensation algorithm is also included in this algorithm to make it applicable in slight moving camera situations. Furthermore, the required parameters can be decided automatically, which can enhance the proposed algorithm to have adaptive threshold ability. It can be integrated into MPEG-4 videophone systems and digital cameras.

In this paper, a novel rate control algorithm suitable for real-time video encoding is proposed. The proposed algorithm uses mean absolute error (MAE) results of motion estimation (ME) to achieve bit-rate control. Neither pre-analysis nor multi-pass encoding is required in our algorithm, which makes real-time hardware implementation possible. A new hardware oriented scene change detection method is also included in this rate control framework to achieve better video quality. Experiments show our rate control algorithm behaves well in all situations. The hardware architecture for this algorithm is also described. Implementation shows our proposed algorithm can be efficiently integrated into a low cost, high efficiency video encoder.

In the new video coding standard, MPEG-4 AVC/JVT/H.264, motion estimation is allowed to use multiple reference frames. The reference software adopts a full search scheme, and the increased computation is in proportion to the number of searched reference frames. However, the reduction of prediction residues is highly dependent on the nature of the sequences, not on the number of searched frames. We present a method to speed up the matching process for multiple reference frames. For each macroblock, we analyze the available information after intra prediction and motion estimation from the previous frame to determine whether it is necessary to search more frames. The information we use includes selected mode, inter prediction residues, intra prediction residues, and motion vectors. Simulation results show that the proposed algorithm can save up to 90% of unnecessary frames while keeping the average miss rate of optimal frames less than 4%.

H.264 is initiated by ITU-T as H.26L and will become a joint standard of ITU-T and MPEG. The coding complexity of H.264 is much higher than MPEG-4 simple profile and advance simple profile algorithms. In order to achieve real-time encoding, hardware implementation is required. The original test model of H.264 (JM) is designed to achieve high coding performance. Some algorithms of the test model require lots of operations with little coding efficiency improvement. And some algorithms create data dependencies that prevent parallel hardware accelerations. This paper presents analysis of the H.264 video coding algorithm from a hardware-oriented viewpoint. Intra prediction, Hadamard transform and motion estimation algorithms are reviewed and modified to a hardware friendly configuration. The rate distortion penalties of these modifications are simulated and shown in this paper.

A motion compensated de-interlacing method with adaptive global motion estimation and compensation is proposed to recover the defects of interlaced video sequence with camera panning, rotating or zooming. GME and GMC are used to recover the change of the whole picture due to camera motions. Two local motion compensated de-interlacing methods are proposed and applied to de-interlace the interlaced video sequences with or without global motion respectively. SAD checking and global/local motion comparator can be used as a block based mode decision system for intra/MC/GMC modes. The proposed algorithm could achieve higher image quality of interlaced video sequences than any other usual de-interlacing algorithm on progressive devices.

This paper presents a cost-effective platform architecture design for MPEG-4 video coding. A fast motion estimator architecture supporting predictive diamond search and spiral full search with halfway termination is implemented to make good compromise between compression performance and design cost. An efficient block-level scheduling for texture coding engine is employed to reduce the hardware cost. Both these key modules are integrated into an efficient platform in hardware/software co-design fashion. With high degree of optimization in both algorithm and architecture levels, a cost-efficient video encoder is implemented. It consumes 256.8 mW at 40 MHz and achieves real-time encoding of 30 CIF (352/spl times/288) frames per second.

An efficient coding system is required for immersive teleconference to transmit stereo video. We propose a novel stereo video coding system by exploiting mesh-based disparity estimation and compensation scheme to achieve high coding efficiency and view synthesis ability. Based on the base-layer-enhance-layer structure, this system can provide stereoscopic scalability and compatibility to standards. With considering asymmetric spatial resolution property, good subjective quality can be achieved in ultra low bitrate situation. A novel fast disparity estimation algorithm named as two-stage iterative block and octagonal matching (TS-IBOM) algorithm is also proposed for this system. Experiments show that the proposed disparity estimation algorithm can generate accurate disparity vectors quickly. It is also shown that the proposed coding system has better coding efficiency than MPEG-4 simple profile and temporal scalability tool. The ultra low bitrate of 69 Kbps can be reached to encode 384/spl times/192 60 fps stereo video.

In this paper, an efficient hardware implementation of two-dimensional shape-adaptive discrete wavelet transform (2-D SA-DWT) with the JPEG2000 defaulted (9,7) filter bank is presented. Two techniques are used to minimize the critical path and the internal buffer size. One technique is the flipping structure which can shorten the critical path of the lifting scheme by flipping multiplier coefficients, rather than pipelining. The other technique is a shape-adaptive boundary handling strategy which can enhance lifting-based architectures to solve boundary extension problems of SA-DWT with little hardware overhead. A prototyping chip of this implementation will be fabricated with TSMC 0.25 /spl mu/m CMOS 1P5M process, and the estimated frequency and the core area are 50 MHz and 2.83 mm/sup 2/, respectively.

Sprite coding is a new objected-based coding technology proposed by MPEG-4 video standard. In this paper, we propose an unsupervised sprite coding system for sport videos, for example, tennis sequences. Our system can provide several important functions. First, the sprite of the background can be generated without any pre-processing masks in our system. Second, it can automatically segment the foreground and background in a video sequence. Third, it can provide the masks of the foreground objects and the tennis ball. The experimental results show that our system has a very good performance and the coding gain of our system compared with MPEG-4 advanced simple profile is 2.5 dB at the low bit rate (270 Kbps) and is 2 dB at the ultra low bit rate (70 Kbps). It can be used in the sport video coding at low bit rate and provides a object-based sport video sequence.

In this paper, a novel rate control algorithm suitable for realtime video encoding is proposed. The proposed algorithm uses mean absolute error (MAE) results of motion estimation (ME) to achieve bitrate control. Neither pre-analysis nor multi-pass encoding is required in our algorithm, which makes realtime hardware implementation possible. A new hardware oriented scene change detection method is also included in this rate control framework to achieve better video quality. Experiment shows our rate control algorithm behaves well in all situations. Hardware architecture for this algorithm is also described. Implementation shows our proposed algorithm can be efficiently integrated into low cost, high efficiency video encoder.

In this work, a novel word-level algorithm of the embedded block coding in JPEG 2000 is proposed. Unlike conventional approaches, all bitplanes of coefficients are processed in parallel in the proposed algorithm. The algorithm is based on the observations and the analysis of the significant state of a coefficient and its contribution to neighbors. As a result of the analysis, there is only the coding pass of the MSB of a coefficient is the required information for the word-level processing. An algorithm for finding the coding pass of the MSB of a coefficient is also proposed. The number of scans of the entire code block is reduced to two scans and is independent of the number of bit planes.

This paper presents an efficient VLSI architecture for the deblocking filter in H.264/JVT/AVC. We use an array of 8/spl times/4 8-bit shift registers with reconfigurable data path to support both horizontal filtering and vertical filtering on the same circuit (a parallel-in parallel-out reconfigurable FIR filter). Two SRAM modules are carefully organized not only for the storage of current macroblock data and adjacent block data but also for the efficient access of pixels in different blocks. Simulation results show that under 0.25 /spl mu/m technology, the synthesized logic gate count is only 19.1 K (not including a 96/spl times/32 SRAM and a 64/spl times/32 SRAM) when the maximum frequency is 100 MHz. Our architecture design can easily support real-time deblocking of 720p (1280/spl times/720) 30 Hz video. It is valuable for platform-based design of H.264 codec.

In the new video coding standard, MPEG-4 AVC/JVT/H.264, motion estimation is allowed to use multiple reference frames. The reference software adopts full search scheme, and the increased computation is in proportion to the number of searched reference frames. However, the reduction of prediction residues is highly dependent on the nature of sequences, not on the number of searched frames. In this paper, we present a method to speed up the matching process for multiple reference frames. For each macroblock, we analyze the available information after intra prediction and motion estimation from previous one frame to determine whether it is necessary to search more frames. The information we use includes selected mode, inter prediction residues, intra prediction residues, and motion vectors. Simulation results show that the proposed algorithm can save up to 90% of unnecessary frames while keeping the average miss rate of optimal frames less than 4%.

H.264 [Committee Draft of Joint Video Specification, July 2002] is initiated by ITU-T as H.26L and will become a joint standard of ITU-T and MPEG. The coding complexity of H.264 is much higher than MPEG-4 simple profile and advance simple profile algorithms. In order to achieve real-time encoding, hardware implementation is required. The original test model of H.264 (JM) [Joint Video Team, August 2002] is designed to achieve high coding performance. Some algorithms of the test model require lots of operations with little coding efficiency improvement. And some algorithms create data dependencies that prevent parallel hardware accelerations. This paper presents analysis of H.264 video coding algorithm in a hardware-oriented viewpoint. Intra prediction, Hadamard transform and motion estimation algorithms are reviewed and modified to a hardware friendly configuration. The rate distortion penalties of these modifications are simulated and shown in this paper.

The watershed transform is a key operator in video segmentation algorithms. However, the computation load of watershed transform is too large for real-time applications. In this paper, a new fast watershed algorithm, named P-watershed, for image sequence segmentation is proposed. By utilizing the temporal coherence property of the video signal, this algorithm updates watersheds instead of searching watersheds in every frame, which can avoid a lot of redundant computation. The watershed process can be accelerated, and the segmentation results are almost the same as those of conventional algorithms. Moreover, an intra-inter watershed scheme (IP-watershed) is also proposed to further improve the results. Experimental results show that this algorithm can save 20%-50% computation without degrading the segmentation results. This algorithm can be combined with any video segmentation algorithm to give more precise segmentation results. An example is also shown by combining a background registration and change-detection-based segmentation algorithm with P-Watershed. This new video segmentation algorithm can give accurate object masks with acceptable computation complexity.

This paper describes an efficient motion estimation algorithm, the predictive line search (PLS), for real-time implementations of MPEG-4 encoder on multimedia processors. The motion-vector predictor is used as the starting point in the search process because the correlation between neighboring motion vectors is strong. The line search pattern is used in the proposed algorithm to reduce the memory access as well as to exploit the special multimedia processor instructions for sum of absolute difference calculations. Experimental results show that the performance of the PLS is very close to that of the full-search (FS) algorithm. Compared with the well-known diamond search and one-dimensional FS, the PLS shows better performance and robustness, especially for high motion sequences. A prototype MPEG-4 encoding system is implemented on a 216-MHz multimedia processor with very long instruction word architecture to verify the effectiveness of the PLS. Real-time encoding of MPEG-4 Simple Profile Level 3 (CIF, 30 fps) can be achieved with only 57% of the processor load.

Embedded block coding with optimized truncation (EBCOT) is the most important technology in the latest image-coding standard, JPEG 2000. The hardware design of the block-coding engine in EBCOT is critical because the operations are bit-level processing and occupy more than half of the computation time of the whole compression process. A general purpose processor (GPP) is, therefore, very inefficient to process these operations. We present detailed analysis and dedicated hardware architecture of the block-coding engine to execute the EBCOT algorithm efficiently. The context formation process in EBCOT is analyzed to get an insight into the characteristics of the operation. A column-based architecture and two speed-up methods, sample skipping (SS) and group-of-column skipping (GOCS), for the context generation are then proposed. As for arithmetic encoder design, the pipeline and look-ahead techniques are used to speed up the processing. It is shown that about 60% of the processing time is reduced compared with sample-based straightforward implementation. A test chip is designed and the simulation results show that it can process 4.6 million pixels image within 1 s, corresponding to 2400 /spl times/ 1800 image size, or CIF (352 /spl times/ 288) 4 : 2 : 0 video sequence with 30 frames per second at 50-MHz working frequency.

Watershed transform is a key operator in video segmentation algorithms. In this paper, a new fast watershed algorithm named P-Watershed for image sequences segmentation is proposed. By utilizing the temporal coherence property of video signal, this algorithm updates watersheds instead of searching watersheds in whole image. The watershed process can be accelerated and the segmentation results are almost the same as those of conventional algorithms. Moreover, an intra-inter watershed scheme (IP-Watershed) is also proposed to further improve the results. Experimental results show this algorithm can save 50% computation without degrading the segmentation results.

This paper presents a new algorithm and architecture for motion estimation. The proposed global elimination algorithm (GEA) is derived from successive elimination algorithm (SEA). The main idea is to remove the branches of SEA to make data flow more regular and suitable for hardware. Besides, the processing time per motion vector for GEA is fixed, no initial guess is required, and the skipping ratio of search positions can be fixed within frames and is even higher than 99%. The average PSNR of compensated frames is almost the same (within 0.1 dB) as that of full-search block matching algorithm (FBMA). An architecture composed of a systolic part, an adder tree, and a comparator tree is also developed for GEA. Simulation results show our design outperforms many FBMA architectures in normalized processing capability per gate and normalized power at gate level.

A novel method to reduce computation of JPEG2000 encoding is proposed. The main concept is that most of the computation in the entropy coder (EBCOT Tier-1) of JPEG2000 is redundancy especially at lower bit-rates, and the proposed method can greatly reduce this redundancy through the feedback of EBCOT Tier-2 processing. By the information generated from Tier-2, the computing time of EBCOT Tier-1 at irreversible wavelet transform mode (9-7 filter) can be reduced to about 40% and 20% at general and high compression rates, respectively. And there is even more reduction at reversible wavelet transform mode (5-3 filter).

Sprite coding, which is a new coding tool in MPEG-4, can achieve high coding efficiency with high subjective quality at low bit rate. Many sprite generation algorithms have been proposed; however, the computational intensity is very high and the quality is not good enough because of the limitation of simple motion models. A novel sprite generation algorithm is proposed with several new techniques. A frame skipping technique can generate the sprite using only several important frames to accelerate the process and achieve similar subjective quality. In addition, boundary matching and multiple sprites techniques can overcome the limitation of simple motion models to achieve high subjective quality with little computation overhead. Experiments show the proposed algorithm is 46 times faster than the algorithms in MPEG-4 VM and have high subjective quality. These techniques can be also applied with other sprite generation algorithms.

Sprite is an image collecting information of a video object through a video sequence. It can be used for efficient video coding, video summary, browsing, and editing. In this paper, three new techniques for sprite generation are proposed. Boundary matching and multiple sprites techniques can improve the subjective quality with refining the positions of the warped frames and generating more than one sprites. The frame skipping technique can skip redundant frames that contain only little new information when the camera revisits a scene several times to accelerate the sprite generation process. Experimental results show that these techniques can be employed independently and can improve the subjective quality as well as reduce the 47.68% - 17.22% runtime of sprite generation. They can be applied with any sprite generation algorithms.

This paper presents a simple but effective algorithm for a pan-tilt-zoom camera to automatically track a single moving object. The proposed tracking algorithm is suitable for but not limited to stationary background, and it can tolerate reasonable noise and light change. The main idea is to initially capture the background information and to calculate the difference between incoming frame and background buffer. Block-based processing is adopted to reduce computation and to alleviate noise effects. Skin-color detection is combined with spatial and temporal information to let the camera more likely to focus on human faces. Even more, the tracking algorithm can be integrated with region-of-interest video coding, which allocates more bits for human faces to produce better subject views. Many practical situations have been tested and the simulation results show that the proposed tracking algorithm is useful for surveillance systems.

For multimedia transmission over noisy channels, the error robustness of JPEG2000 evidently outperforms that of JPEG. Since JPEG2000 is based on the discrete wavelet transform (DWT), traditional error concealment algorithms for still images in the discrete cosine transform (DCT) domain are not suitable for JPEG2000. In JPEG2000, decoding is processed bitplane by bitplane. Any data loss occurring in the bitstream will affect the consequent bitplanes and their wavelet coefficients. To solve this problem, the JPEG2000 VM7.2 program replaces the missing wavelet coefficients by zeros. However, the replacement may affect lots of significant nonzero coefficients such that some high frequency components are lost. In this paper, we present a novel error concealment algorithm for image transmission in the bitplane base. The proposed algorithm recovers the damaged bitplane data according to the cross subband and undamaged bitplane information. The recovered wavelet coefficients are similar with error-free data. The objective results show that the proposed algorithm has 3/spl sim/8dB improvement than those without the error resilient mechanism. From a subjective viewpoint, the proposed algorithm can achieve much smoother edges on the reconstructed image using our concealment algorithm.

In this paper, a novel video transmission architecture is proposed to meet the low-delay and error robust requirement of wireless video communications. This architecture uses FEC coding and ARQ protocol to provide efficient bandwidth access from wireless link. In order to reduce ARQ delay, a video proxy server is implemented at the base station. This video proxy not only reduces the ARQ response time but also provides error tracking functionality. We use H.263 as the experimental platform for our architecture. Experiments show that average luminance PSNR decreases only 0.35db for the "Foreman" sequence under a random error condition of 10/sup -3/ error probability.

This paper presents an efficient VLSI architecture design of MPEG-4 shape coding, which is the key technology for supporting the content-based functionality of the MPEG-4 video standard. The real-time constraint of MPEG-4 shape coding leads to a heavy computational bottleneck on today's computer architectures. To overcome this problem, design analysis and optimization of MPEG-4 shape coding are addressed. By utilizing the RISC-based model, computational behaviors of the MPEG-4 shape coding tool are carefully examined and analyzed. The characteristic of a large amount of bit-level data processing and data transfer of MPEG-4 shape coding motivates the optimization of bit-level data operations. Applying data-flow optimization and data reuse techniques, bit-level computation-efficient architectures, such as data-dispatch-based binary-shaped motion estimation, the delay-line model, and configurable context-based arithmetic coding, are designed to accelerate bit-level processing. These hardware blocks are integrated and scheduled in a very efficient data flow to achieve real-time performance for MPEG-4 CPL2 (core profile level 2) specification at 23.5 MHz clock rate. The system architecture is implemented using Verilog HDL and synthesized with a 0.35 /spl mu/m four-layer CMOS standard library.

An efficient moving object segmentation algorithm suitable for real-time content-based multimedia communication systems is proposed in this paper. First, a background registration technique is used to construct a reliable background image from the accumulated frame difference information. The moving object region is then separated from the background region by comparing the current frame with the constructed background image. Finally, a post-processing step is applied on the obtained object mask to remove noise regions and to smooth the object boundary. In situations where object shadows appear in the background region, a pre-processing gradient filter is applied on the input image to reduce the shadow effect. In order to meet the real-time requirement, no computationally intensive operation is included in this method. Moreover, the implementation is optimized using parallel processing and a processing speed of 25 QCIF fps can be achieved on a personal computer with a 450-MHz Pentium III processor. Good segmentation performance is demonstrated by the simulation results.

Video communications over wireless networks often suffer from various errors. A novel video transmission architecture is proposed to meet the low-delay and error-robust requirement of wireless video communications. This architecture uses forward error correction coding and automatic repeat request (ARQ) protocol to provide efficient bandwidth access from wireless link. In order to reduce ARQ delay, a video proxy server is implemented at the base station. This video proxy not only reduces the ARQ response time, but also provides error-tracking functionality. The complexity of this video proxy server is analyzed. Experiment shows that about 8.9% of the total macroblocks need to be transcoded under a random-error condition of 10/sup -3/ error probability. Because H.263 is the most popular video coding standard for video communication, we use it as an experiment platform. A data-partition scheme is also used to enhance error-resilience performance. This architecture is also suitable for various motion-compensation-based standards like H.261, H.263 series, MPEG-1, MPEG-2, MPEG-4, and H.264. For "Foreman" sequence under a random-error condition of 10/sup -3/ error probability, luminance peak signal-to-noise ratio decreases only 0.35 dB, on average.

Abstract- A complete, low cost baseline JPEG encoder soft IP and its chip implementation are presented in this paper. It features user-defined, run-time re-configurable quantization tables, highly modularized and fully pipelined architecture. A prototype, synthesized with COMPASS cell library, has been implemented in TSMC 0.6-pm single-poly, triple-metal process. It can run up to 40 MHz at 3.3V. This IP can be easily integrated into various application systems, such as scanner, PC camera and color FAX, etc. I.

This paper proposes a new cost-effective architecture for mathematical morphology named partial-result-reuse (PRR) architecture. For many real-time applications of mathematical morphology, the hardware implementation is necessary; however, the hardware cost of most existing morphology architectures is too high when dealing with large structuring elements. With the partial-result-reuse concept and self-affinity property of general structuring elements, the proposed architecture is more cost-effective and more general than the existing morphology architectures. It can deal with morphological operations with arbitrary structuring elements and can be used for other semi-group operations, and only 2[log/sub 2/n] comparators are needed for n/spl times/n structuring elements. Simulation shows that this architecture can dramatically reduce the hardware cost of morphological operations with all kinds of structuring elements.

Data partitioning, one of the MPEG-4 video error resilience tools, enables better error robustness. However, it suffers from serious error propagation problem. In this paper, we use an experimental approach to model the error propagation with several commonly used error detection conditions. It is shown that errors detected in forward section of texture data may be propagated from motion data, while those in DCT coefficients mostly result from themselves. Furthermore, the motion marker is the major error source for several error conditions detected in motion part. According to these characteristics, motion marker assumption and backtracking-based concealment strategies are proposed to achieve more accurate error localization in bitstream domain. 1.

Analysis and architecture design of the key modules in JPEG2000 are presented in this paper. For Discrete Wavelet Transform (DWT), a lifting based DWT core for the default 5-3 and 9-7 filters in part I of JPEG2000 is proposed. Folded architecture is adopted in DWT to reduce the hardware cost and to achieve the higher hardware utilization. For Embedded Block Coding with Optimized Truncation (EBCOT), column-based coding architecture of Tier-1 block coding engine is proposed. The context formation efficiency is increased by adopting two speedup methods. The computation cycle of the block coding engine is reduced to about 40 % of previous work. 1.

A fast watershed based video segmentation algorithm is proposed in this paper. Watershed based video segmentation algorithms are the mainstream because of high subjective quality; however, they are too computational intensive for real-time applications. Combining prior fast segmentation algorithm, which is based on change detection and background registration, and predictive watershed, a novel fast watershed algorithm for image sequences segmentation, this algorithm can give precise segmentation results and achieve real-time requirement. Simulation shows processing speed of 9.4 QCIF fps can be achieved on a PC with a Pentium-III 800 MHz processor.

In this paper, a low-power full-search block matching (FSBM) motion-estimation design for the ITU-T recommendation H.263+ standard is proposed. New motion-estimation modes in H.263+ can be fully supported by our architecture. Unlike most previously presented motion-estimation chips, this design can deal with 8/spl times/8 and 16/spl times/16 block size with different searching ranges. Basically, the proposed architecture is composed of an integer pixel unit with 64 processing elements, and a half-pixel unit with interpolation, a control unit, and data registers. In order to minimize power consumption, gated-clock and dual-supply voltages are used. This design has been realized by TSMC 0.6 /spl mu/m SPTM CMOS technology. The power consumption is 423.8 mW at 60 MHz and the throughput is 36 fps in CIF format.

This paper proposes an efficient architecture for the two-dimensional discrete wavelet transform (2-D DWT). The proposed architecture includes a transform module, a RAM module, and a multiplexer. In the transform module, we employ the polyphase decomposition technique and the coefficient folding technique to the decimation filters of stages 1 and 2, respectively. In comparison with other 2-D DWT architectures, the advantages of the proposed architecture are 100% hardware utilization, fast computing time (0.5-0.67 times of the parallel filters'), regular data flow, and low control complexity, making this architecture suitable for next generation image compression systems, e.g., JPEG-2000.

This paper presents various analyses of computational behavior. Namely, the number of datapath operations and memory access on the core profile level 2 (CPL2) of MPEG-4 video standard. These analyzed data exploit the load distribution and mode selection of the video system. The exploration of data-flow behavior and its derived computation of MPEG-4 video processing algorithms will then drive through an efficient architecture design.

In this paper, the hardware-oriented structure analysis and an efficient and flexible bitstream parser for MPEG-4 video are presented. The analysis of bitstream structure explores processing requirement and design constraint for bitstream-level processing. The proposed architecture is basically RAM-based that can be reconfigured for various applications. For high bitrate as about 40 Mbit/s, it needs only about 19 MIPS to parse the bitstream. The impact of the proposed architecture on MPEG-4 video is to enhance and extend the processing for bit domain translation and related real time applications.

Many VLSI architectures for computing the discrete wavelet transform (DWT) were presented, but the parallel input data sequence and the programmability of the 2-D DWT were rarely mentioned. In this paper, we present a parallel-processing VLSI architecture to compute the programmable 2-D DWT, including various wavelet filter lengths and various wavelet transform levels. The proposed architecture is very regular and easy to extend. To eliminate high frequency components, the pixel values outside the boundary of the image are mirror-extended as the symmetric wavelet transform and the mirror-extension is realized via the routing network. This design has been implemented and fabricated in a 0.35 /spl mu/m 1P4M CMOS technology and the working frequency is 50 MHz. The chip size is about 5200 /spl mu/m/spl times/2500 /spl mu/m. For a 256/spl times/256 image, the chip can perform 30 frames per second with the filter length varying from 2 to 20 and with various levels. Hence the proposed architecture is suitable for real-time applications such as JPEG 2000.

In this paper, a low power 8/spl times/8 2D DCT architecture based on direct 2D approach is proposed. The direct 2D consideration reduces computational complexity. According to this algorithm, a parallel distributed arithmetic (DA) architecture at reduced supply voltage is derived. In the real circuit implementation of the chip, a hybrid-architecture adder of low power consumption is designed, as well as a power-saving ROM and a low voltage two-port SRAM with sequential access. The resultant 2D DCT chip is realized by 0.6 /spl mu/m single-poly double-metal technology. Critical path simulation indicates a maximum input rate of 133 MHz, and it consumes 138 mW at 100 MHz.

This paper describes a data-interlacing architecture with two-dimensional (2-D) data-reuse for full-search blockmatching algorithm. Based on a one-dimensional processing element (PE) array and two data-interlacing shift-register arrays, the proposed architecture can efficiently reuse data to decrease external memory accesses and save the pin counts. It also achieves 100% hardware utilization and a high throughput rate. In addition, the same chips can be cascaded for different block sizes, search ranges, and pixel rates.

This paper describes a data-interlacing architecture with two-dimensional (2-D) data-reuse for full-search block-matching algorithm. Based on some cascading strategies, the same chips can be flexibly cascaded for different block sizes, search ranges, and pixel rates. In addition, the cascading chips can efficiently reuse data to decrease external memory accesses and achieve a high throughput rate. Our results demonstrate that the architecture with 2-D data-reuse is a flexible, low-pin-counts, high-throughput, and cascadable solution for full search block-matching algorithm.

A new error resilience algorithm for recovering the lost blocks or motion vectors is presented. It combines the overlapped motion compensation and the side match criterion to make the effect of lost blocks subjectively imperceptible. The side match criterion takes advantage of the spatial contiguity and interpixel correlation of image to select the best-fit replacement among the motion vectors of spatially contiguous candidate blocks. Particularly, to mask the blocking artifacts, we incorporate an overlapping technique to create a subjectively closer approximation to the true error-free image.

This paper describes a data-interlacing architecture with two-dimensional (2-D) data-reuse for full-search block-matching algorithm. Based on a one-dimensional processing element (PE) array and two data-interlacing shift-register arrays, the proposed architecture can efficiently reuse data to decrease external memory accesses and save the pin counts. It also achieves 100% hardware utilization and a high throughput rate. In addition, the same chips can be cascaded for different block sizes, search ranges, and pixel rates.

Most video sequence coding systems use block motion compensation to remove temporal redundancy for video compression due to the regularity and simplicity. A new error concealment algorithm for recovering the lost or erroneously received motion vectors is presented. It combines the overlapped motion compensation and the side match criterion to make the effect of lost motion vectors subjectively imperceptible. The side match criterion takes advantage of the spatial contiguity and interpixel correlation of image to select the best-fit replacement among the motion vectors of spatially contiguous candidate blocks. Particularly, to mask the blocking artifacts, we incorporate an overlapping technique to create a subjectively closer approximation to the true error-free image.

Among the various transform techniques for image compression, the discrete cosine transform (DCT) is the most popular and effective one in practical image and video coding applications, such as high-definition television (HDTV). We develop a novel 8/spl times/8 two-dimensional (2-D) discrete cosine transform/inverse discrete cosine transform (DCT/IDCT) architecture based on the direct 2-D approach and the rotation technique. The computational complexity is reduced by taking advantage of the special attribute of a complex number. Both the parallel and the folded architectures are proposed. Unlike other approaches, the proposed architecture is regular and economically allowable for VLSI implementation. Compared to the row-column method, less internal wordlength is needed in order to meet the error requirement of IDCT, and the throughput of the proposed architecture can achieve two times that of the row-column method with 30% hardware increased.

In this paper, a new video coding algorithm suitable for the very low bit rate video application is presented. It takes advantage of the prior knowledge of the image type to segment the image into different regions, then codes each region with different coding criterion and method according to the different importance. An adaptive region-classified vector quantization strategy is also exploited in this algorithm. With the segmentation of the frame and high correlation between frames, better codebooks of vector quantisation are constructed to improve the quality. According to the simulation results, acceptable quality at about 10 kbit/s can be obtained for the typical test sequences.

A modified optical flow algorithm (MOFA) is proposed for the development of a very low bit-rate video coding system. Another edge preserving constraint and pyramid approach are suggested to generate an accurate motion field and reduce the possibility of being trapped in a local minimum. Post-processing schemes are also designed to eliminated the problems for special regions. Compared with other motion estimation algorithms, the proposed method gives a more exact estimation in terms of the PSNR and subjective view. The arbitrarily shaped transform of the motion field is then selected to further remove the spatial redundancy. According to some primary simulation results, most of the test sequences are compressed into 16 kbps or lower with excellent picture quality.

An intelligent self-adjusting weighted median filter for removing impulsive noise in images is presented. Three main techniques are developed to implement this self-adjusting weighted median filter: an intelligent classification to divide the image data into the "corrupted" pixels and the "uncorrupted" pixels, an efficient algorithm to find the median output of any weight set, and a realistic training procedure without the noise free image to obtain the proper weights. Our simulations on some test images demonstrate that the proposed filter has less smoothing effect and smaller mean square error (MSE) or mean absolute error (MAE) measurement than the standard median filter. At the same time, the quality of the filter output has been enhanced significantly. Finally, a demonstration chip of this weighted median filter for the 5/spl times/5 window size is also presented.

A pseudo object-oriented video coding system is proposed and implemented. In order to increase the coding efficiency, a cache VQ algorithm is suggested to further compress those areas where motion estimation fails. According to our primary simulation results, the visual quality of long-timed sequences is still acceptable even for bit-rates below 10 kbps. In addition to the high compression ratio for very low bit-rates, content-based applications are also expected since the proposed system utilizes the segmented motion field; furthermore, the occurrences of prediction errors generally locate at emotionally important parts, e.g. eyes and mouth, etc. All the coded components are not only useful for compression but also meaningful for video recognition.

Since three-dimensional (3-D) subband coding has been introduced, most researches on 3-D subband coding perform temporal filtering first. In this paper, however, we investigate the best permutation strategy for temporal, vertical, and horizontal filtering to minimize the requirement of delay elements and find that the results are opposite to our expectation.

A scalable implementation scheme for multirate finite impulse response (FIR) filters is presented in this paper. The implementation scheme can reduce the time and area cost of a multirate FIR filter simultaneously. According to the throughput requirement, the proposed scheme achieves a minimum-area design with specified speed. Furthermore, by employing the scalable implementation scheme, we also propose an efficient design technique for subband filter banks.

No abstract available.

A novel and efficient block-matching motion estimation criterion called minimized maximum error (MiniMax) is considered. The proposed method can save hardware area about 15% with acceptable video performance. A chip which combines the MiniMax matching criterion and the one-dimensional full search algorithm is presented. The ASIC is motivated by the need of the intensive computational demand to perform motion estimation in real time. The proposed single chip can match the applications of H.261 and MPEG international standards. Chip cascading is allowed for larger searching range applications.>

We propose a modified fast algorithm for discrete cosine transform (DCT) by transferring the results from the discrete Hartley transform (DHT) to one additional CORDIC (coordinate rotation digital computer) computing stage. A fast CORDIC-based systolic array is designed with four derived attractive features, including: (1) the single/double data folding feature; (2) the constructive feature; (3) the to-computing feature; and (4) the redundant computation. Due to its properties, the proposed design has an efficient hardware utilization and a high throughput rate. By using the redundant path, this design also has the capability of error detection.>

In this paper, we propose a modified Malvar algorithm that can transfer the Discrete Hartley Transform (DHT) results to the Discrete Cosine Transform (DCT) by using a one additional CORDIC computing stage. A fast CORDIC-based systolic array is designed with four attractive features the single/double data folding feature; the constructive feature; the co-computing feature; and redundant path computation. Due to the existence of the redundant path, the proposed design also has the capability of error detection.>

To design a 32-bit logarithmic number system (LNS) processor, this paper presents two novel techniques: Digit-Partition (DP) to design log/sub 2/(1.x) function and Iterative Difference by Linear Approximation (IDLA) to design 2/sup 0.x/ function. The experimental result reveals that the proposed design is more attractive than the previous researches in the LNS processor.>

This paper describes fully pipelined parallel architectures for the 3-step search block-matching motion estimation algorithm. Difficulties of this algorithm in hardware implementation were overcome by use of intelligent data arrangement and memory configuration. Techniques for reducing interconnections and external memory accesses were also developed. Because of their low costs, high speeds, and low memory bandwidth requirements, the proposed architectures provide efficient solutions for real-time motion estimations required by various video applications.>

A tree-structure based architecture for vector quantization (VQ) is proposed and implemented by VLSI in this paper, The proposed folded-tree architecture is designed by 0.8 /spl mu/m CMOS VLSI technology. The die size is 4.65/spl times/5.21 mm/sup 2/ and estimated clock rate is about 34 MHz, which satisfies most real-time applications. Since it is basically a mean squared error computation circuit, various kinds of VQ algorithms can apply on the proposed architecture.>

In this paper, we propose a binary-tree structure neural network model suitable for structured clustering. During and after training, the centroids of the clusters in this model always form a binary tree in the input pattern space. This model is used to design tree search vector quantization codebooks for image coding. Simulation results show that the acquired codebook not only produces better-quality images but also achieves a higher compression ratio than conventional tree search vector quantization. When source coding is applied after VQ, the new model performs better than the generalized Lloyd algorithm in terms of distortion, bits per pixel, and encoding complexity for low-detail and medium-detail images.>

A new hardware-oriented algorithm called the one-dimensional full search (1DFS) is presented for block-matching motion estimation in video compression. The simulation for this algorithm follows H.261 and MPEG international standards. In MPEG simulation, structures with 1-, 2- and 3-frame interpolation are compared. The performance of 1DFS is superior to that of other fast search algorithms. And it has more regular data flow, data reuse and less control overhead. It is an alternative for 2D full search block matching and achieves a good compromise between computational complexity and performance. With competent performance and reasonable computation complexity, the proposed method is more suitable for real-time hardware realization of a VLSI motion estimator for video applications.>

A technique for video data compression which combines optimized block truncation coding (OBTC) and a conventional motion compensation (MC) algorithm is presented. It has been shown that the proposed OBTC based system is much better than the conventional system in terms of implementation, complexity, and accuracy. Four benchmark images and two benchmark sequences are used as test examples.>

Proposes three modifications on the 3-step hierarchical search for video coding: the multiple-candidate search that improves the estimation accuracy, a method of subsampling that reduces computation and input data amount, and an overlapping strategy to improve the accuracy of a large-area search. Experimental results show that combining these techniques provides high-speed and high-precision motion estimators with reduced on-chip buffers and lower input bandwidth requirements.>

The paper describes fully pipelined parallel architectures for the 3-step hierarchical search block-matching algorithm, a fast motion estimation algorithm for video coding. The advantage of this algorithm was completely utilized by use of intelligent data arrangement and memory configuration. Techniques for reducing interconnections and external memory accesses were also developed. Because of their low costs, high speeds, and low memory bandwidth requirements, the proposed 3-PE, 9-PE, and 27-PE architectures provide efficient solutions for real-time motion estimations required by video applications of various data rates, from low bit-rate video to HDTV systems.>

This paper presents a rate-optimal scheduling for real-time DSP algorithms. By using pipelining and unfolding techniques, the parallel characteristics of recursive DSP algorithms can be exploited. A novel unfolding technique is developed to unravel all concurrency in the recursive data flow graph. A perfect rate unfolded data flow graph is also introduced, which can cause a fully static rate optimal functional pipeline schedule. Experimental results have shown that the proposed method can always yield rate-optimal designs with a smaller unfolding factor compared to previous studies.>

A new approach for hardware verification using symbolic state transition graphs (implemented in BDDs) is presented. We propose a novel idea, a symbolic state transition graph (STG), which can represent finite state machines FSM in terms of the relations between symbolic input variables, state variables, and output results, rather than the exact input-output bit patterns. Compared to conventional STG methods, the symbolic STG is more concise, higher-level, has fewer states and is easier to specify. Based on the transition relation method and an event-driven scheduling technique to compute the symbolic states, we propose two algorithms to verify the circuit implementation with respect to its symbolic STGs. The algorithms can be used to find out a necessary condition for the implementation to satisfy the specification, which can be used for the allocation of design errors.>

A building-block design for VLSI-based array dividers with concurrent error detection by recomputing using partitioned architecture (REPA) is proposed. The basic concept is that the divide array can be divided into two identical parts and its operation can be completed by using one part through two iterative calculations. With two such parts, a concurrent error detection scheme based on a space redundancy approach can be designed, and error detection is achieved at each iteration. The design is better than previous designs in terms of area requirement, time penalty, error detection capability, and detection period. Advanced analysis with m partitions is included. The experimental results are attractive, especially for certain designs with application-specified tradeoffs between speed performance and area cost.>

A set of techniques for rate-optimal scheduling of recursive DSP algorithms is presented. The underlying microprocessor system is based on the RISC architecture. The retiming process evenly distributes the delays and optimizes the scheduling time of each iteration. The unfolding of recursive data flow graph exploits intraiteration and interiteration parallelism, and the schedule can be more compacted. It is shown that after evenly retiming and unfolding with the optimal unfolding factor, a fully-extended precedence graph is generated, which can be scheduled in rate-optimal by a simple contiguous scheduling. The optimal unfolding factor is equal to the delay counts of the critical loop.>

An efficient block matching algorithm (BMA) for motion estimation is presented. This BMA exploits motion correlation of neighbor blocks in the temporal direction to reduce the search area. Instead of finding the 2D motion vector directly, this BMA finds two 1D displacements in parallel on two axes independently within the reduced search area. Simulation results show that this algorithm can rival conventional BMAs for performance. The hardware-oriented features of this BMA guarantee that it is more suitable for the hardware realization of a VLSI motion estimator.>

The authors present an efficient block-matching algorithm called the parallel hierarchical one-dimensional search (PHODS) for motion estimation. Instead of finding the two-dimensional motion vector directly, the PHODS finds two one-dimensional displacements in parallel on the two axes (say x and y) independently within the search area. The major feature of this algorithm is that its search speed for the motion vector is faster than that of the other search algorithms on account of its simpler computations and parallelism. Compared with previous research in terms of four measurements, the PHODS can rival those algorithms for performance. The hardware-oriented features of the PHODS, i.e., regularity, simplicity, and parallelism, guarantee that the PHODS is more suitable for hardware realization of a VLSI motion estimator.>