This paper presents evaluation results of Low Complexity Enhancement Video Codec (LCEVC) using its reference implementation (LTM-5.4.1) when combined with HEVC (HM-16.25) and VVC (VTM-19.0) as base layer codecs. Tests were performed for 4K content, following JVET Random Access (RA) Common Testing Conditions (CTC). The anchor is the result of single layer coding, i.e., encoding 4K videos by VTM or HM only, and the test is the result of base layer codec, which encodes quarter-resolution version of 4K videos by using VTM or HM as base layer, followed by enhancement layer coded using L TM-5.4.1, where enhancement layer is obtained as a difference between original4K video and upsampled reconstructed frames from the base layer. PSNR BD-rate was used for objective evaluation. PSNR results show that LCEVC is introducing significant losses. Also, a viewing session was performed to evaluate visual quality of LCEVC when combined with VTM, which showed that LCEVC does not provide any visual benefits for 4K content.

The Software Engineering (SE) course primarily teaches students how to develop and maintain high-quality software using systematic processes, methods, and tools. Five teaching issues in the SE course, including course breadth, limited teaching assistant resources, limited teaching interaction, learning weaknesses, and poor team communication, hinder students from absorbing technical concepts, knowledge, and skills for necessary subjects in the SE field. In recent years, Digital Interactive Learning has received increasing attention. Among them, Chatbot was significantly under the spotlight. It can confirm the user's intention by analyzing the content of the dialogue and simulating a human conversation. To address the teaching issues for the SE course and leverage the benefits brought by Chatbots, in this study, we designed a Chatbot system, called TABot, to assist teaching and learning, especially for the SE course. This study designed multiple features that support students' learning, such as frequently asked questions, requests for course materials, and adaptive quiz practices. Furthermore, TABot also provides team supporting functions, such as reading the user requirements of their team project, group member contribution analysis, and GitHub commit retrieval. After being applied to an SE course, 80% of the students agreed that TABot could help them learn better for the SE course.

Versatile Video Coding (VVC) is the latest video coding standard jointly developed by ITU-T VCEG and ISO/IEC MPEG. In this paper, technical details and experimental results for the VVC block partitioning structure are provided. Among all the new technical aspects of VVC, the block partitioning structure is identified as one of the most substantial changes relative to the previous video coding standards and provides the most significant coding gains. The new partitioning structure is designed using a more flexible scheme. Each coding tree unit (CTU) is either treated as one coding unit or split into multiple coding units by one or more recursive quaternary tree partitions followed by one or more recursive multi-type tree splits. The latter can be horizontal binary tree split, vertical binary tree split, horizontal ternary tree split, or vertical ternary tree split. A CTU dual tree for intra-coded slices is described on top of the new block partitioning structure, allowing separate coding trees for luma and chroma. Also, a new way of handling picture boundaries is presented. Additionally, to reduce hardware decoder complexity, virtual pipeline data unit constraints are introduced, which forbid certain multi-type tree splits. Finally, a local dual tree is described, which reduces the number of small chroma intra blocks.

This paper provides an overview of the coding algorithms of the Joint Exploration Model (JEM) for video compression with capability beyond HEVC, which was developed by the Joint Video Exploration Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). The goal of the JEM development and experimentation was to provide evidence that sufficient coding efficiency improvement over the High Efficiency Video Coding (HEVC) standard can be achieved, which would justify the need for a new video coding standard with a compression capability significantly exceeding that of HEVC. The development of the JEM provided an ability to conduct studies toward that goal in a verifiable and collaborative manner and led to the launching of the project to develop the new Versatile Video Coding (VVC) standard. Objective metric gains exceeding 30% were measured for most of the tested high-resolution video content that represents current demanding new applications, and subjective testing using human observers showed even more benefit.

This paper describes a video coding technology proposal submitted by MediaTek in April 2018 in response to a joint call for proposals (CfP) issued by ITU-T VCEG and ISO/IEC MPEG. The proposal uses the conventional block-based hybrid coding approach with a breakthrough design of block structure, substantial improvements of coding tools, and new inventions of loop filtering techniques. First, the block structure is designed using a highly flexible partitioning scheme, where each coding tree unit in a picture is split into multiple coding units (CUs) by a recursive quaternary tree followed by a recursive binary-ternary tree. The prediction unit (PU) and transform unit (TU) concepts are unified with the CU concept, i.e., PU and TU are basically the same as CU. Moreover, inter prediction, intra prediction, transform, quantization, in-loop filtering, and entropy coding are all improved. Finally, new loop filtering techniques are invented, where the proposed convolutional neural network loop filtering is of particular interest. The proposed video codec achieves 43.81% average bit rate savings based on objective measures compared to the High Efficiency Video Coding (HEVC) anchors and is among the best-performing CfP responses both objectively and subjectively. It has been partially adopted into the working draft of the Versatile Video Coding (VVC) standard.

Non-orthogonal multiple access (NOMA) is a promising technology for future mobile networks due to superior spectrum efficiency. In this paper, we model the interaction between the base station and multiple users as a Stackelberg game and devise a fast resource allocation method consisting of resource block allocation and power allocation. The objective is to serve as many users as possible at rates beyond their requirements, while seeking to enhance total revenue. We derive a closed-form formula for optimal power allocation. Besides, we convert the resource block allocation problem into the problem of finding a maximum weight independent set for a chordal graph, which only takes linear time. Simulation results show that the fast method we propose outperforms existing algorithms in total throughput and the number of users whose rate demands are attained.

BACKGROUND: High-throughput sequencing offers higher throughput and lower cost for sequencing a genome. However, sequencing errors, including mismatches and indels, may be produced during sequencing. Because, errors may reduce the accuracy of subsequent de novo assembly, error correction is necessary prior to assembly. However, existing correction methods still face trade-offs among correction power, accuracy, and speed. RESULTS: We develop a novel overlap-based error correction algorithm using FM-index (called FMOE). FMOE first identifies overlapping reads by aligning a query read simultaneously against multiple reads compressed by FM-index. Subsequently, sequencing errors are corrected by k-mer voting from overlapping reads only. The experimental results indicate that FMOE has highest correction power with comparable accuracy and speed. Our algorithm performs better in long-read than short-read datasets when compared with others. The assembly results indicated different algorithms has its own strength and weakness, whereas FMOE is good for long or good-quality reads. CONCLUSIONS: FMOE is freely available at https://github.com/ythuang0522/FMOC .

The emerging standards for screen content coding (SCC) and 3D video coding known as High Efficiency Video Coding (HEVC)-SCC and 3D-HEVC, respectively, have been developed as extensions of HEVC. Compared with camera-captured video, screen content video and depth maps usually contain significantly fewer colors or sample intensities and much sharper edges in a frame. In this paper, a new coding method named segmental prediction is proposed to code screen content and depth maps more efficiently. First, samples in a prediction block are classified into several segments according to the sample intensities. Second, a particular predictor is calculated for each segment. Then an offset for a segment is derived at the encoder side and signaled to the decoder side. Finally, a single sample value is assigned to each segment in the reconstructed block. The experimental results show that the proposed approach can achieve a 2.2% average Bjontegaard Distortion-rate (BD-rate) saving under the HEVC-SCC common test conditions (CTCs) for sequences of the category YUV, text and graphics with motion, 1080p, with the all intra configuration. It can also achieve a 0.6% average BD-rate saving for synthesized views under the 3D-HEVC CTCs with the random access configuration.

Palette mode is the new coding tool that has been adopted in the Screen Content Coding Extensions of High Efficiency Video Coding (HEVC SCC). Palette mode can represent colour clusters for screen content efficiently and can be summarized into two parts: palette coding tools and colour index map coding tools. This paper proposes two techniques to improve colour index map coding: transition copy and prediction across coding unit boundary. The former is for exploring the correlation between an index and all of the previously coded indices; the latter for utilizing the correlation between the content along the coding unit boundary. Experimental results reportedly show that, compared with HEVC SCC, the proposed techniques can achieve 3.2%, 2.2%, and 1.7% BD-rate savings compared with HEVC SCC for “YUV, text & graphics with motion, 1080p & 720p sequences” under all intra, random access, and low-delay B common test conditions, respectively.

Palette mode, as a new coding tool, is adopted in the Screen Content Coding Extensions of High Efficiency Video Coding (HEVC SCC) that is being defined by the Joint Collaborative Team on Video Coding (JCT-VC). In the palette mode, pixels in a coding unit (CU) are represented by selected representative colours according to the characteristics of screen contents in which pixel values usually concentrate on few colour values. This paper introduces the palette mode in HEVC SCC, and our contributions to the palette mode are also presented. Experimental results show that disabling the palette mode suffers 20.3%, 13.0%, and 7.5% BD-rate increases for “YUV, text & graphics with motion, 1080p & 720p sequences” under all intra (AI), random access (RA), and low-delay B (LB) common test conditions, respectively.

As the state-of-the-art video coding standard, High Efficiency Video Coding (HEVC) has been finalized in January 2013. To support the coding of multiple views and associated depth data, the development of the HEVC extension for 3D video coding is also about to be finalized. New coding unit (CU) level coding tools are added to the HEVC design to improve the compression capabilities for both video views and depth data. In this paper, we introduce a new coding mode termed as “Single Depth Intra Coding Mode” to efficiently code the smooth area within a depth map. The concept of single depth mode is to simply reconstruct a coding block with a single depth value based on block merging scheme under the HEVC quad-tree based block partitioning. The proposed single depth mode has been adopted into the working draft and 3D-HEVC test model 12 (HTM-12.0). The experimental results evaluated under the common test conditions (CTC) for 3D-HEVC show that the proposed technique achieves up to 5% bit-rate saving for depth coding and on average 0.3% bit-rate saving regarding the synthesis results over the texture and depth bit-rate with decoding time reduced by 4%.

The 3D extension of High Efficiency Video Coding (3D-HEVC) is a new international video coding standard that has been developed by the Joint Collaborative Team on 3D Video Coding Extensions (JCT-3V). It aims at improving the coding efficiency of 3D and multi-view videos by introducing new coding tools to utilize the correlations between views and between texture and depth components. In this paper, we propose an inter-view motion prediction (inter-view merge candidate) and an inter-component motion prediction (texture merge candidate) to explore the inter-view and the inter-component redundancies. The proposed schemes were adopted in 3D-HEVC and the experimental results demonstrate that the proposed inter-view merge candidate and texture merge candidate achieve significant BD-rate reductions of 20% for dependent texture views and 9% for the synthesized texture views under the common test conditions used for 3D-HEVC standardization.

The great debate over the Sounding Balloon Project has fast become a hot topic around the world recently. This project can acquire a number of photos from the edge of space, and research the atmosphere which sustains human existence. It is very important for weather forecasting and upper-air sounding. However, due to the instability of the wind, we cannot forecast the track of the balloon. No matter whether the equipment lands in valleys, on roadways or the seas, retrieving the equipment is full of challenges. In this paper, we propose a state-of-the-art method which combines the sounding balloon with an Unmanned Aerial Vehicle (UAV). On the one hand, the sounding balloon may reach altitudes of 30 km. On the other hand, the UAV can achieve autonomous flight. We combine the advantages of both, enabling the sounding balloon to carry the equipment to high altitude, and the UAV to deliver our payload to the designated location landing. In our method, we use a self-developed App to control the UAV, to record the sensor values through built-in sensors, to take the photographs, and so on. This allows our project to proceed more smoothly. According to our experiments, we achieve our aims of safe and effective retrieval, he successful acquisition of lots of photos and information from the high altitude, and an in depth analysis. This integration of software and hardware opens up a different application from that of the past and offers excellent accomplishments.

The target of 3D video coding is to compress Multiview Video plus Depth (MVD) format data, which consist of a texture image and its corresponding depth map. In the MVD format, the depth map plays an important role for successful services in 3D video applications, because it enables the user to experience 3D by generating arbitrary intermediate views. The depth map has a strong correlation with its associated texture data, so it can be utilized to improve texture coding efficiency. This paper introduces a novel and efficient depth-based texture coding scheme. It includes depth-based motion vector prediction, block-based view synthesis prediction, and adaptive luminance compensation, which were adopted in an AVC-compatible 3D video coding standard. Simulation results demonstrate that the proposed scheme reduces the total coding bitrates of texture and depth by 19.06% for the coded PSNR and 17.01% for the synthesized PSNR in a P-I-P view prediction structure, respectively.

In this paper, we present a resource allocation algorithm for OFDMA-based systems which handles both real-time and non-real-time traffic. For real-time traffic, the QoS requirements are specified with delay bound and loss probability. The resource allocation problem is formulated as one which maximizes system throughput subject to the constraint that the bandwidth allocated to a flow is no less than its minimum requested bandwidth, a value computed based on loss probability requirement and running loss probability. A user-level proportional-loss scheduler is adopted to determine the resource share for flows attached to the same subscriber station (SS). In case the available resource is not sufficient to provide every flow its minimum requested bandwidth, we maximize the amount of real-time traffic transmitted subject to the constraint that the bandwidth allocated to an SS is no greater than the sum of minimum requested bandwidths of all flows attached to it. Moreover, a pre-processor is added to maximize the number of real-time flows attached to each SS that meet their QoS requirements. We show that, in any frame, the proposed proportional-loss scheduler guarantees QoS if there is any scheduler which guarantees QoS. Simulation results reveal that our proposed algorithm performs better than previous works.

This paper provides a technical overview of a newly added in-loop filtering technique, sample adaptive offset (SAO), in High Efficiency Video Coding (HEVC). The key idea of SAO is to reduce sample distortion by first classifying reconstructed samples into different categories, obtaining an offset for each category, and then adding the offset to each sample of the category. The offset of each category is properly calculated at the encoder and explicitly signaled to the decoder for reducing sample distortion effectively, while the classification of each sample is performed at both the encoder and the decoder for saving side information significantly. To achieve low latency of only one coding tree unit (CTU), a CTU-based syntax design is specified to adapt SAO parameters for each CTU. A CTU-based optimization algorithm can be used to derive SAO parameters of each CTU, and the SAO parameters of the CTU are inter leaved into the slice data. It is reported that SAO achieves on average 3.5% BD-rate reduction and up to 23.5% BD-rate reduction with less than 1% encoding time increase and about 2.5% decoding time increase under common test conditions of HEVC reference software version 8.0.

Recently, an efficient coding tool named adaptive interpolation filtering (AIF) has been proposed to hybrid video coding scheme. By introducing Wiener filter into the fractional-pixel interpolation procedure, AIF can reduce the inter-prediction error and improve coding efficiency significantly. However, the training-based Wiener filter mechanism brings AIF an inherent multi-pass encoding structure, which imposes big burdens on the encoder in terms of huge computational complexity and memory access. In this paper, we propose a single-pass-based localized adaptive interpolation filtering (SPL-AIF) algorithm for video coding, which can reduce the complexity of AIF dramatically without sacrifice of its outstanding coding performance. The proposed SPL-AIF algorithm is based on the observation that there is a high correlation among optimal interpolation filters of consecutive frames, and different regions in a frame often possess different statistical characteristics. Accordingly, the proposed algorithm can be designed including two major parts. First, a competitive filter set which includes the optimal interpolation filters of several previous frames as well as the fixed H.264/AVC interpolation filters is built up for the coding of the current frame. Then a rate-distortion optimization criterion is used to select the best one at macroblock (MB) level. In order to reduce overhead, a predictive coding method is used to compress the filter signaling flag for each MB. Experimental results show that, by using the proposed algorithm, the encoding complexity can be reduced significantly while the average coding gain in Bjöntegaard distortion bit-rate reduction can be improved about 1% compared with the multi-pass AIF. The proposed method has been adopted into the Video Coding Expert Group Key Technology Area software.

A new video coding tool, sample adaptive offset (SAO), is introduced in this paper. SAO has been adopted into the Working Draft of the new video coding standard, High-Efficiency Video Coding (HEVC). The SAO is located after deblocking in the video coding loop. The concept of SAO is to classify reconstructed pixels into different categories and then reduce the distortion by simply adding an offset for each category of pixels. The pixel intensity and edge properties are used for pixel classification. To further improve the coding efficiency, a picture can be divided into regions for localization of offset parameters. Simulation results show that SAO can achieve on average 2% bit rate reduction and up to 6% bit rate reduction. The run time increases for encoders and decoders are only 2%.

High Efficiency Video Coding (HEVC) is a new international video coding standard that has been developed by the Joint Collaborative Team on Video Coding (JCT-VC). In this paper, an overview of the motion vector coding techniques for HEVC is presented. Our three proposed coding tools for the motion vector predictor (MVP) in the Inter, Skip and Merge modes of HEVC are also presented, which includes a new location of the temporal MVP, a priority-based derivation method of spatial MVPs, and a derivation method of temporal MVPs. A combination of these three tools can achieve on average 1.3%, 1.8%, 1.2% and 2.2% bit rate reductions for high efficiency random access, low complexity random access, high efficiency low delay, and low complexity low delay, respectively.

In this paper, a one-pass encoding algorithm is proposed for adaptive loop filter (ALF) in high-efficiency video coding (HEVC). ALF can improve both subjective and objective video quality, but it also requires a lot of encoding passes (i.e. picture buffer accesses) that will significantly increase external memory access, encoding latency, and power consumption. Therefore, we propose a method to estimate filtering distortion without performing real filter operation. The number of encoding passes can be effectively reduced from 16 to 1. Combined with an initial guess of filter-on/off blocks by using time-delayed filters, the proposed one-pass algorithm only induces average 0.17% BD-rate increase.

An adaptive interpolation filtering (AIF) algorithm has been proposed to improve the conventional hybrid video coding scheme recently. Although such an algorithm does improve coding efficiency significantly, its encoding complexity, in terms of computational complexity and memory access, increases dramatically due to its inherent two-pass encoding structure. In this paper, we propose a novel single-pass based algorithm which can reduce the complexity of AIF significantly. In our method, a competitive filter set which includes optimal filters trained from several previous frames and the fixed H.264 filters is considered for the current coding frame. A rate-distortion optimization (RDO) criterion is then used to select the best one at macro-block (MB) level. In order to reduce overhead, a predictive coding method is used to compress the filter type for each MB. Experimental results show that, by using the proposed algorithm, the encoding complexity can be significantly reduced without sacrifice of coding gain.

Adaptive interpolation filtering (AIF) algorithms have been proposed to enhance the hybrid video coding scheme recently. Although these algorithms can improve coding efficiency significantly, their encoders suffer huge increase in complexity in terms of latency and memory access due to its inherent multi-pass encoding procedure. In this paper, we present a novel single-pass solution for these algorithms, which allows optimal selection among different interpolation filters. In this solution, time-delayed interpolation filters are used to achieve single-pass encoding, and localized ratedistortion (RD) selection is used to compensate the possible coding loss from time-delayed filters. Experimental results show that the proposed method is efficient for All AIF techniques in current ITU-T/SG16 reference software. By using the proposed method, single-pass encoding with multiple AIF filters can be achieved while maintaining similar coding efficiency as multi-pass AIF.

Piggyback is an effective way to reduce the MAC protocol overhead of wireless networks. This paper considers the physical data rate and transmission bit error to derive the criterion of piggyback. Numerical results show that the improvement of piggyback can reach 40% for some cases. For given the bit error rate and the data rate, we also provide the criterion for every combination in a table.

The advanced MIMO (multiple-input multiple-output) system can provide higher capacity gain or higher diversity gain in modern wireless networks. The capacity gain is achieved by interference cancellation technique to fully utilize the spatial degree of freedom. In a multiuser wireless system, the increased degrees of freedom provide the chance to select multiple wireless stations (STAs) to maximize the throughput. In this paper, we consider a scenario where the access point (AP) equipped with two antennas to simultaneously communicate with two STAs, each with single antenna. After obtaining the statuses of the size of buffered traffic and the achievable transmission rate, the AP should pair compatible STAs to minimize the loss of spatial degree of freedom. Therefore, three pairing algorithms are proposed and are evaluated through computer simulations. The algorithms have much less implementation complexity than the brute force pairing algorithm and achieve satisfactory performance of throughput.

For QoS requirements of real-time traffic, IEEE 802.11 working group introduces a QoS-aware channel access mechanism, called hybrid coordination function (HCF), which consists of contention-based enhanced distributed channel access (EDCA) and contention-free HCF controlled channel access (HCCA). The TXOP allocation and the admission control units of the HCCA reference scheduler are only appropriate for constant bit rate (CBR) flows. It may result in serious packet loss for variable bit rate (VBR) flows. In this paper, we propose a simple admission control algorithm which adopts Gaussian distribution to approximate VBR traffic. Numerical results obtained from computer simulations show that our proposed algorithm can effectively and efficiently allocate transmission opportunity (TXOP) durations to QoS-enhanced stations (QSTAs) to guarantee a predefined packet loss probability. Moreover, our proposed scheme can easily handle multiple VBR flows of the same QSTA to get the advantage of multiplexing gain.

Due to the multiple reference frame motion estimation (MRF-ME), an H.264/AVC encoder requires ultrahigh memory bandwidth. Conventional multiple reference frames single current macroblock (MRSC) scheme only considers the data reuse within one frame, and requires on-chip memory size and off-chip memory bandwidth in proportional to the reference frame number. In this paper, a single reference frame multiple current macroblocks (SRMC) scheme is presented to further exploit the data reuse at frame level. With frame-level rescheduling of the motion estimation ME procedures in different reference frames, one loaded search window can be utilized by multiple current MBs in different original frames. The demanded on-chip memory size and off-chip memory bandwidth for MRF-ME can thus be reduced to those supporting only one reference frame. Moreover, based on SRMC scheme, an architecture prototype with two-stage mode decision flow is proposed. For HDTV specifications, 62.21 KB (74.8%) of SRAM and 364.3 MB/s (62.6%) of system bandwidth are saved in comparison with the MRSC scheme

H.264/AVC significantly outperforms previous video coding standards with many new coding tools. However, the better performance comes at the price of the extraordinarily huge computational complexity and memory access requirement, which makes it difficult to design a hardwired encoder for real-time applications. In addition, due to the complex, sequential, and highly data-dependent characteristics of the essential algorithms in H.264/AVC, both the pipelining and the parallel processing techniques are constrained to be employed. The hardware utilization and throughput are also decreased because of the block/MB/frame-level reconstruction loops. In this paper, we describe our techniques to design the H.264/AVC video encoder for HDTV applications. On the system design level, in consideration of the characteristics of the key components and the reconstruction loops, the four-stage macroblock pipelined system architecture is first proposed with an efficient scheduling and memory hierarchy. On the module design level, the design considerations of the significant modules are addressed followed by the hardware architectures, including low-bandwidth integer motion estimation, parallel fractional motion estimation, reconfigurable intrapredictor generator, dual-buffer block-pipelined entropy coder, and deblocking filter. With these techniques, the prototype chip of the efficient H.264/AVC encoder is implemented with 922.8 K logic gates and 34.72-KB SRAM at 108-MHz operation frequency.

In the new video coding standard H.264/AVC, motion estimation (ME) is allowed to search multiple reference frames. Therefore, the required computation is highly increased, and it is in proportion to the number of searched reference frames. However, the reduction in prediction residues is mostly dependent on the nature of sequences, not on the number of searched frames. Sometimes the prediction residues can be greatly reduced, but frequently a lot of computation is wasted without achieving any better coding performance. In this paper, we propose a context-based adaptive method to speed up the multiple reference frames ME. Statistical analysis is first applied to the available information for each macroblock (MB) after intra-prediction and inter-prediction from the previous frame. Context-based adaptive criteria are then derived to determine whether it is necessary to search more reference frames. The reference frame selection criteria are related to selected MB modes, inter-prediction residues, intra-prediction residues, motion vectors of subpartitioned blocks, and quantization parameters. Many available standard video sequences are tested as examples. The simulation results show that the proposed algorithm can maintain competitively the same video quality as exhaustive search of multiple reference frames. Meanwhile, 76 %-96 % of computation for searching unnecessary reference frames can be avoided. Moreover, our fast reference frame selection is orthogonal to conventional fast block matching algorithms, and they can be easily combined to achieve further efficient implementations.

A computation-aware motion estimation algorithm is proposed in this paper. Its goal is to find the best block-matching results in a computation-limited and computation-variant environment. Our algorithm is characterized by a one-pass flow with adaptive search strategy. In the prior scheme, Tsai et al. propose that all macroblocks are processed simultaneously, and more computation is allocated to the macroblock with the largest distortion among the entire frame in a step-by-step fashion. This implies that random access of macroblocks is required, and the related information of neighboring macroblocks cannot be used to be prediction. The random access flow requires a huge memory size for all macroblocks to store the up-to-date minimum distortions, best motion vectors, and searching steps. On the contrary, our one-pass flow processes the macroblocks one by one, which can not only significantly reduce the memory size but also effectively utilize the context information of neighboring macroblocks to achieve faster speed and better quality. Moreover, in order to improve the video quality when the computation resource is still sufficient, the search pattern is allowed to adaptively change from diamond search to three step search, and then to full search. Last but not least, traditional block matching speed-up methods are also combined to provide much better computation-distortion curves

The paper summarizes our design experiences of various image and video codec IPs. The design issues and methodology of custom video codecs are discussed. The design methodology can be summarized as four stages: system analysis; algorithm optimization; architecture exploration; code development. Based on these guidelines, several design cases are presented, including the proposed JPEG, MPEG-4, and H.264 architectures.

This paper provides a survey of state-of-the-art hardware architectures for image and video coding. Fundamental design issues are discussed with particular emphasis on efficient dedicated implementation. Hardware architectures for MPEG-4 video coding and JPEG 2000 still image coding are reviewed as design examples, and special approaches exploited to improve efficiency are identified. Further perspectives are also presented to address the challenges of hardware architecture design for advanced image and video coding in the future.

Intra prediction with rate-distortion constrained mode decision is the most important technology in H.264/AVC intra frame coder, which is competitive with the latest image coding standard JPEG2000, in terms of both coding performance and computational complexity. The predictor generation engine for intra prediction and the transform engine for mode decision are critical because the operations require a lot of memory access and occupy 80% of the computation time of the entire intra compression process. A low cost general purpose processor cannot process these operations in real time. In this paper, we proposed two solutions for platform-based design of H.264/AVC intra frame coder. One solution is a software implementation targeted at low-end applications. Context-based decimation of unlikely candidates, subsampling of matching operations, bit-width truncation to reduce the computations, and interleaved full-search/partial-search strategy to stop the error propagation and to maintain the image quality, are proposed and combined as our fast algorithm. Experimental results show that our method can reduce 60% of the computation used for intra prediction and mode decision while keeping the peak signal-to-noise ratio degradation less than 0.3 dB. The other solution is a hardware accelerator targeted at high-end applications. After comprehensive analysis of instructions and exploration of parallelism, we proposed our system architecture with four-parallel intra prediction and mode decision to enhance the processing capability. Hadamard-based mode decision is modified as discrete cosine transform-based version to reduce 40% of memory access. Two-stage macroblock pipelining is also proposed to double the processing speed and hardware utilization. The other features of our design are reconfigurable predictor generator supporting all of the 13 intra prediction modes, parallel multitransform and inverse transform engine, and CAVLC bitstream engine. A prototype chip is fabricated with TSMC 0.25-/spl mu/m CMOS 1P5M technology. Simulation results show that our implementation can process 16 mega-pixels (4096/spl times/4096) within 1 s, or namely 720/spl times/480 4:2:0 30 Hz video in real time, at the operating frequency of 54 MHz. The transistor count is 429 K, and the core size is only 1.855/spl times/1.885 mm/sup 2/.

We contributed a new VLSI architecture for fractional motion estimation of the H.264/AVC video compression standard. Seven inter-related loops extracted from the complex procedure are analyzed and two decomposing techniques are proposed to parallelize the algorithm for hardware with a regular schedule and full utilization. The proposed architecture, also characterized by a reusable feature, can support situations in different specifications, multiple standards, fast algorithms and some cost considerations. H.264/AVC baseline profile level 3 with complete Lagrangian mode decision can be realized with 290K gates at operating frequency of 100 MHz. It is a useful intellectual property (IP) design for platform based multimedia systems.

The critical path of the hardware for the global elimination algorithm (GEA) is too long to meet the real-time constraints for high-end applications. In this paper, we propose a new parallel GEA and its corresponding architecture. By dividing candidate blocks into independent groups and finding the most probable candidates of each group in parallel, instead of sequentially searching within the whole search range, parallel design can be developed as an array of GEA processing elements with much shorter critical path. Besides, the GEA processing element is optimized to reduce 30% of the gates, and the 2D data reuse is organized to save 80% of the SRAM bandwidth, which also reduces a lot of power. Simulation results show that our implementation can achieve real time processing of D1 30 Hz video with search range as H[-64, +63.5] V[-32, +31.5] while the operating frequency is 70 MHz, and the gate count is 113 K. Compared with full search, our gate count is six times smaller under the same frequency, and the PSNR loss is at most 0.1-0.2 dB.

This paper presents a new block matching motion estimation algorithm and its VLSI architecture design. The proposed global elimination algorithm (GEA) was derived from successive elimination algorithm (SEA), which can skip unnecessary sum of absolute difference (SAD) calculation by comparing minimum SAD with subsampled SAD (SSAD). Our basic idea is to separate the decision of early termination and SAD calculation for each candidate block to make data flow more regular and suitable for hardware. In short, we first compare the rough characteristics of all candidate blocks with the current block (SSAD). In turn, we select several best roughly matched candidate blocks to re-compare them with the current block by using detailed characteristics (SAD). Other features of GEA include fixed processing cycles, no initial guess, and high video quality (almost the same as full search). Unlike other fast algorithms, the mapping of GEA to hardware is very simple. We proposed an architecture that is composed of a systolic part to efficiently compute SSAD, an adder tree to support both SSAD and SAD calculations, and a comparator tree to avoid expensive sorting circuits. Simulation results show that our design is much more area efficient than many full-search architectures while maintaining high video quality and processing capability.

Automatic video segmentation plays an important role in real-time MPEG-4 encoding systems. Several video segmentation algorithms have been proposed; however, most of them are not suitable for real-time applications because of high computation load and many parameters needed to be set in advance. This paper presents a fast video segmentation algorithm for MPEG-4 camera systems. With change detection and background registration techniques, this algorithm can give satisfying segmentation results with low computation load. The processing speed of 40 QCIF frames per second can be achieved on a personal computer with an 800 MHz Pentium-III processor. Besides, it has shadow cancellation mode, which can deal with light changing effect and shadow effect. A fast global motion compensation algorithm is also included in this algorithm to make it applicable in slight moving camera situations. Furthermore, the required parameters can be decided automatically, which can enhance the proposed algorithm to have adaptive threshold ability. It can be integrated into MPEG-4 videophone systems and digital cameras.

In the new video coding standard, MPEG-4 AVC/JVT/H.264, motion estimation is allowed to use multiple reference frames. The reference software adopts a full search scheme, and the increased computation is in proportion to the number of searched reference frames. However, the reduction of prediction residues is highly dependent on the nature of the sequences, not on the number of searched frames. We present a method to speed up the matching process for multiple reference frames. For each macroblock, we analyze the available information after intra prediction and motion estimation from the previous frame to determine whether it is necessary to search more frames. The information we use includes selected mode, inter prediction residues, intra prediction residues, and motion vectors. Simulation results show that the proposed algorithm can save up to 90% of unnecessary frames while keeping the average miss rate of optimal frames less than 4%.

H.264 is initiated by ITU-T as H.26L and will become a joint standard of ITU-T and MPEG. The coding complexity of H.264 is much higher than MPEG-4 simple profile and advance simple profile algorithms. In order to achieve real-time encoding, hardware implementation is required. The original test model of H.264 (JM) is designed to achieve high coding performance. Some algorithms of the test model require lots of operations with little coding efficiency improvement. And some algorithms create data dependencies that prevent parallel hardware accelerations. This paper presents analysis of the H.264 video coding algorithm from a hardware-oriented viewpoint. Intra prediction, Hadamard transform and motion estimation algorithms are reviewed and modified to a hardware friendly configuration. The rate distortion penalties of these modifications are simulated and shown in this paper.

Sprite coding is a new objected-based coding technology proposed by MPEG-4 video standard. In this paper, we propose an unsupervised sprite coding system for sport videos, for example, tennis sequences. Our system can provide several important functions. First, the sprite of the background can be generated without any pre-processing masks in our system. Second, it can automatically segment the foreground and background in a video sequence. Third, it can provide the masks of the foreground objects and the tennis ball. The experimental results show that our system has a very good performance and the coding gain of our system compared with MPEG-4 advanced simple profile is 2.5 dB at the low bit rate (270 Kbps) and is 2 dB at the ultra low bit rate (70 Kbps). It can be used in the sport video coding at low bit rate and provides a object-based sport video sequence.

This paper presents an efficient VLSI architecture for the deblocking filter in H.264/JVT/AVC. We use an array of 8/spl times/4 8-bit shift registers with reconfigurable data path to support both horizontal filtering and vertical filtering on the same circuit (a parallel-in parallel-out reconfigurable FIR filter). Two SRAM modules are carefully organized not only for the storage of current macroblock data and adjacent block data but also for the efficient access of pixels in different blocks. Simulation results show that under 0.25 /spl mu/m technology, the synthesized logic gate count is only 19.1 K (not including a 96/spl times/32 SRAM and a 64/spl times/32 SRAM) when the maximum frequency is 100 MHz. Our architecture design can easily support real-time deblocking of 720p (1280/spl times/720) 30 Hz video. It is valuable for platform-based design of H.264 codec.

In the new video coding standard, MPEG-4 AVC/JVT/H.264, motion estimation is allowed to use multiple reference frames. The reference software adopts full search scheme, and the increased computation is in proportion to the number of searched reference frames. However, the reduction of prediction residues is highly dependent on the nature of sequences, not on the number of searched frames. In this paper, we present a method to speed up the matching process for multiple reference frames. For each macroblock, we analyze the available information after intra prediction and motion estimation from previous one frame to determine whether it is necessary to search more frames. The information we use includes selected mode, inter prediction residues, intra prediction residues, and motion vectors. Simulation results show that the proposed algorithm can save up to 90% of unnecessary frames while keeping the average miss rate of optimal frames less than 4%.

H.264 [Committee Draft of Joint Video Specification, July 2002] is initiated by ITU-T as H.26L and will become a joint standard of ITU-T and MPEG. The coding complexity of H.264 is much higher than MPEG-4 simple profile and advance simple profile algorithms. In order to achieve real-time encoding, hardware implementation is required. The original test model of H.264 (JM) [Joint Video Team, August 2002] is designed to achieve high coding performance. Some algorithms of the test model require lots of operations with little coding efficiency improvement. And some algorithms create data dependencies that prevent parallel hardware accelerations. This paper presents analysis of H.264 video coding algorithm in a hardware-oriented viewpoint. Intra prediction, Hadamard transform and motion estimation algorithms are reviewed and modified to a hardware friendly configuration. The rate distortion penalties of these modifications are simulated and shown in this paper.

The watershed transform is a key operator in video segmentation algorithms. However, the computation load of watershed transform is too large for real-time applications. In this paper, a new fast watershed algorithm, named P-watershed, for image sequence segmentation is proposed. By utilizing the temporal coherence property of the video signal, this algorithm updates watersheds instead of searching watersheds in every frame, which can avoid a lot of redundant computation. The watershed process can be accelerated, and the segmentation results are almost the same as those of conventional algorithms. Moreover, an intra-inter watershed scheme (IP-watershed) is also proposed to further improve the results. Experimental results show that this algorithm can save 20%-50% computation without degrading the segmentation results. This algorithm can be combined with any video segmentation algorithm to give more precise segmentation results. An example is also shown by combining a background registration and change-detection-based segmentation algorithm with P-Watershed. This new video segmentation algorithm can give accurate object masks with acceptable computation complexity.

This paper describes an efficient motion estimation algorithm, the predictive line search (PLS), for real-time implementations of MPEG-4 encoder on multimedia processors. The motion-vector predictor is used as the starting point in the search process because the correlation between neighboring motion vectors is strong. The line search pattern is used in the proposed algorithm to reduce the memory access as well as to exploit the special multimedia processor instructions for sum of absolute difference calculations. Experimental results show that the performance of the PLS is very close to that of the full-search (FS) algorithm. Compared with the well-known diamond search and one-dimensional FS, the PLS shows better performance and robustness, especially for high motion sequences. A prototype MPEG-4 encoding system is implemented on a 216-MHz multimedia processor with very long instruction word architecture to verify the effectiveness of the PLS. Real-time encoding of MPEG-4 Simple Profile Level 3 (CIF, 30 fps) can be achieved with only 57% of the processor load.

Watershed transform is a key operator in video segmentation algorithms. In this paper, a new fast watershed algorithm named P-Watershed for image sequences segmentation is proposed. By utilizing the temporal coherence property of video signal, this algorithm updates watersheds instead of searching watersheds in whole image. The watershed process can be accelerated and the segmentation results are almost the same as those of conventional algorithms. Moreover, an intra-inter watershed scheme (IP-Watershed) is also proposed to further improve the results. Experimental results show this algorithm can save 50% computation without degrading the segmentation results.

This paper presents a new algorithm and architecture for motion estimation. The proposed global elimination algorithm (GEA) is derived from successive elimination algorithm (SEA). The main idea is to remove the branches of SEA to make data flow more regular and suitable for hardware. Besides, the processing time per motion vector for GEA is fixed, no initial guess is required, and the skipping ratio of search positions can be fixed within frames and is even higher than 99%. The average PSNR of compensated frames is almost the same (within 0.1 dB) as that of full-search block matching algorithm (FBMA). An architecture composed of a systolic part, an adder tree, and a comparator tree is also developed for GEA. Simulation results show our design outperforms many FBMA architectures in normalized processing capability per gate and normalized power at gate level.

Sprite coding, which is a new coding tool in MPEG-4, can achieve high coding efficiency with high subjective quality at low bit rate. Many sprite generation algorithms have been proposed; however, the computational intensity is very high and the quality is not good enough because of the limitation of simple motion models. A novel sprite generation algorithm is proposed with several new techniques. A frame skipping technique can generate the sprite using only several important frames to accelerate the process and achieve similar subjective quality. In addition, boundary matching and multiple sprites techniques can overcome the limitation of simple motion models to achieve high subjective quality with little computation overhead. Experiments show the proposed algorithm is 46 times faster than the algorithms in MPEG-4 VM and have high subjective quality. These techniques can be also applied with other sprite generation algorithms.

Sprite is an image collecting information of a video object through a video sequence. It can be used for efficient video coding, video summary, browsing, and editing. In this paper, three new techniques for sprite generation are proposed. Boundary matching and multiple sprites techniques can improve the subjective quality with refining the positions of the warped frames and generating more than one sprites. The frame skipping technique can skip redundant frames that contain only little new information when the camera revisits a scene several times to accelerate the sprite generation process. Experimental results show that these techniques can be employed independently and can improve the subjective quality as well as reduce the 47.68% - 17.22% runtime of sprite generation. They can be applied with any sprite generation algorithms.

This paper presents a simple but effective algorithm for a pan-tilt-zoom camera to automatically track a single moving object. The proposed tracking algorithm is suitable for but not limited to stationary background, and it can tolerate reasonable noise and light change. The main idea is to initially capture the background information and to calculate the difference between incoming frame and background buffer. Block-based processing is adopted to reduce computation and to alleviate noise effects. Skin-color detection is combined with spatial and temporal information to let the camera more likely to focus on human faces. Even more, the tracking algorithm can be integrated with region-of-interest video coding, which allocates more bits for human faces to produce better subject views. Many practical situations have been tested and the simulation results show that the proposed tracking algorithm is useful for surveillance systems.

A fast watershed based video segmentation algorithm is proposed in this paper. Watershed based video segmentation algorithms are the mainstream because of high subjective quality; however, they are too computational intensive for real-time applications. Combining prior fast segmentation algorithm, which is based on change detection and background registration, and predictive watershed, a novel fast watershed algorithm for image sequences segmentation, this algorithm can give precise segmentation results and achieve real-time requirement. Simulation shows processing speed of 9.4 QCIF fps can be achieved on a PC with a Pentium-III 800 MHz processor.