High-Efficiency Video Coding (HEVC) is the latest video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). An effective rate control mechanism plays a critical role in making the best possible use of the network resources. The coding performance can be maximised through the appropriate allocation of bits under the constraints of a total bit rate budget and the buffer size. Existing rate control strategies generally determine the rate control parameters according to previous encoding results. However, there is no priori-knowledge available when the first frame of a video sequence is encoded or scene changes occur. Consequently, it is challenging to choose appropriate rate control parameters in such cases. In order to address this issue, we investigated deep learning oriented methods and proposed a Convolutional Neural Network (CNN)-based rate control scheme for HEVC intra-coding. As the CNN has the advantage of local perception and the Coding Tree Unit's (CTU's) Quantisation Parameter (QP) value is determined during the CTU level rate control process, we applied the CNN at the CTU level as well to achieve an accurate QP prediction. An improved QP determination model (R-QP model) involving only two model coefficients (µ and v) was suggested to simplify the QP calculation process. A CNN was used to predict the model coefficients, namely µ and v. Our CNN has four convolution layers, two pooling layers and three fully connected layers. Each of the convolution layers contains a Rectified Linear Unit (ReLU). The last fully connected layer outputs the predicted values of µ and v. The images from the UCID dataset and the RAISE dataset were used to train the CNN. In order to obtain a universal model, none of the frames in the HEVC common test sequences were employed as training images. In order to achieve a reasonable bit budget allocation, a CNN-based framework was used to predict the bit consumption for each CTU according to the entire picture content that it contains. At the training stage, the network took the original images and the QP values as inputs, and the corresponding actual bit consumptions were used as the labels. Once the CNN coefficients have been obtained through training, the bit consumption for each CTU under different QP values can be directly predicted from the original images. The predicted bit consumption was subsequently used to readjust the bit budget for each CTU to achieve a more reasonable bit allocation. We implemented the proposed CNN-based rate control scheme in the HEVC reference software HM 16.9. As the priori-knowledge of the previous encoding results had not been considered in the proposed algorithm, only the first frame of each video sequence was encoded. The experimental results showed that our proposed scheme performs better than the default rate control implementation in the HEVC reference software, namely the BDBR is reduced by 1.33% on average while maintaining the same reconstructed picture quality. Compared with the state-of-the-art CNN-based rate control algorithm proposed by Li, our schemes achieved better coding efficiency. It can be concluded that the proposed R-QP model is effective in describing the rate-distortion relationship in HEVC, and the CNN-based bit allocation scheme is appropriate for ensuring that the best use of each bit is made.

Scalable High-efficiency Video Coding (Scalable HEVC, SHVC) is the most recent scalable video coding standard, which is developed by the Joint Collaborative Team on Video Coding (JCT-VC). In order to meet the needs of various client terminals, network conditions, and user demands, a multi-layer encoding framework is employed to generate the scalable bitstream. In addition to the desirable scalabilities, SHVC also aims to achieve better coding efficiency than previous standards. Therefore, all the advanced coding tools ofHEVC are inherited, and several additional tools are specifically designed to support scalability. However, the superior coding efficiency is achieved at the cost of significantly increased computational complexity. Given the fact that an SHVC encoder is considerably more complex than an HEVC encoder, this makes a great demand of a fast SHVC encoder that incurs only a negligible degradation in compression efficiency. In order to address the above issue, several fast schemes have been suggested for SHVC intra and inter coding. However, temporal, spatial and inter-layer correlations have not been fully exploited. In addition, most existing algorithms aimed only at the improvement either on the intra coding process or the inter coding process. If all the available information including the texture complexity, the motion activity, as well as different types of correlation can be appropriately utilised, it is expected that the computational complexity can be reduced further. Therefore, a fast Coding Unit (CU) size decision algorithm is proposed for spatial SHVC intra coding to early terminate the CU size decision process. The energy distribution is exploited to classify the texture complexity, and the available information from the BL is used to narrow the coding depth level ranges for the LCUs in the EL. Unlikely intra modes are removed by considering the inter-layer, spatial and temporal correlations. Different candidate mode lists are constructed for the PUs of various sizes. In addition, the implementation after RMD is also improved. A fast Prediction Unit (PU) prediction mode decision method is also suggested for SHVC inter coding, in which the motion complexity and the inter-layer dependency of PU modes are jointly used to determine the best PU prediction mode for the PU in the EL. The proposed algorithms were implemented in the SHVC reference software SHM 12.0. Two-layers encoding structure was employed for spatial scalability, and the scalability factor was 2. Common test conditions were satisfied. The All Intra (AI) and Lowdelay configurations were employed to evaluate the proposed fast intra and inter coding algorithms. The quantisation parameters (QPs) for the BL and EL were (22,20), (27,25), (32,30) and (37,35). Experimental results showed that the proposed fast intra algorithm reduces the encoding time by 48% on average, while the proposed fast inter algorithm achieves an average time reduction of more than 36% over the unchanged SHM encoder, and the loss in Rate-Distortion (RD) performance is acceptable. It can be concluded that the proposed algorithms outperform the algorithms previously proposed.

Scalable High Efficiency Video Coding (SHVC) provides high compression efficiency at the expense of considerable computational complexity. In this paper, a fast algorithm is proposed to reduce the computational complexity of SHVC intra coding. The coding depth information, texture complexity, and spatio-temporal correlation are jointly used to achieve a faster depth decision process. The mode dependency between the base layer and the enhancement layer is combined with the temporal correlation to simplify the mode decision process. Experimental results demonstrate that the proposed scheme saves encoding time by up to 59% compared with the SHM12.0 encoder with negligible degradation in Rate Distortion (RD).

In High Efficiency Video Coding (HEVC), a quad-tree based Coding Unit (CU) partitioning scheme is employed, achieving a substantial improvement in coding efficiency compared with previous standards. The superior coding efficiency of HEVC is achieved at the expense of greatly increased complexity. A fast intra coding scheme consisting of fast CU depth decision and fast prediction mode decision is proposed to reduce the computational requirement. Classification of the homogeneity of video content using an adaptive double thresholds scheme is employed to reduce the number of Rate Distortion (RD) evaluations. The partition information of spatially neighbouring CUs is utilised to further narrow the depth range. The construction of the initial candidate list is improved for each Prediction Unit (PU). Then the prediction mode correlation between neighbouring quad-tree coding levels is considered to predict the most likely coding mode. The Hadamard cost of prediction modes is examined to further reduce the candidate modes. The computational complexity of HEVC intra coding is therefore reduced. Simulation results show that the proposed algorithm reduces encoding time by up to 71% compared with the HM 13.0 implementation, while having a negligible impact on rate distortion, with increases in bit-rate of 1.82%.

HEVC employs a quad-tree based Coding Unit (CU) structure to achieve a significant improvement in coding efficiency compared with previous standards. However, the computational complexity is greatly increased. We proposed a fast mode decision algorithm to reduce intra coding complexity. Firstly, an initial candidate list of intra modes is constructed for each Prediction Unit (PU). The prediction mode correlation between adjacent quad-tree coding levels and between temporal neighbouring frames is used to predict the most likely coding mode. The number of prediction mode that need to be evaluated in residual quad-tree (RQT) process is further reduced by taking the Hadamard cost of prediction mode into consideration. Simulation results show that the proposed algorithm saves encoding time by up to 51% compared with the HM 13.0 implementation, while having a negligible impact on rate distortion.

High Efficiency Video Coding (HEVC) incorporates a flexible quad-tree block partitioning scheme and up to 35 prediction modes for intra coding. This enables a significant improvement in coding efficiency compared with previous standards. The superior coding efficiency of HEVC is achieved at the expense of greatly increased complexity. A fast Coding Unit (CU) depth decision algorithm is proposed to reduce the computational requirement for intra coding. An adaptive double thresholds scheme is employed to classify the homogeneity of video content. The classification is used to reduce the number of Rate Distortion (RD) evaluations in the CU depth decision process. The partition information of the temporally co-located CU and the spatially neighbouring CUs is jointly utilised to further narrow the depth range that needs to be evaluated. The computational complexity of HEVC intra coding is therefore reduced. Simulation results show that the proposed algorithm reduces encoding time by up to 57% compared with the HM 13.0 implementation, while having a negligible impact on rate distortion, with PSNR losses of 0.01dB and increases in bit-rate of 0.31%.

An improved rate control algorithm for the Scalable Video Coding (SVC) extension of H.264/AVC is described. The rate control scheme applied to the Base Layer (BL) of SVC adopts the linear Mean Absolute Difference (MAD) prediction and quadratic Rate Distortion (RD) models inherited from H.264/AVC. A MAD prediction error always exists and cannot be avoided. However, some encoding results of the base layer can be used to inform the coding of the enhancement layers (ELs), thus benefitting from the bottom-up coding structure of SVC. This property forms the basis for the proposed rate control approach. Simulation results show that accurate rate control is achieved and, compared to the default rate control algorithm of SVC, namely the JVT-G012 rate control scheme, the average PSNR is increased by 0.27dB or the average bit rate is reduced by 4.81%.

In Scalable Video Coding (SVC), inter-layer prediction is utilized to improve the coding efficiency of the enhancement layers. However the rate control scheme in the Joint Scalable Video Model (JSVM) software lacks consideration of the implications of inter-layer prediction as it was designed for non-scalable video encoders. In this work, a novel rate control algorithm is proposed for when inter-layer prediction is employed in SVC. Firstly, a Rate-Quantization (R-Q) model for inter-layer prediction coding of the spatial enhancement layers is developed. Secondly, an optimized Mean Absolute Difference (MAD) prediction model for spatial enhancement layers is proposed, that considers the MAD from previous temporal frames and previous spatial frames together. Simulation results show that rate control accuracy is maintained to within 0.07% on average. Compared with the default rate control algorithm employed in SVC, namely the JVT-G012 rate control scheme, the proposed algorithm improves the average PSNR by up to 0.26dB or produces an average saving in bit rate of up to 4.66%.

A fast mode decision algorithm for efficient implementation of the scalable video coding (SVC) extension of H.264/AVC is described. SVC incorporates interlayer prediction, a new tool that exploits as much lower layer information as possible in order to improve the coding efficiency of the enhancement layer. However, it also greatly increases the computational complexity. A fast mode selection algorithm that exploits the correlation of a macroblock in the enhancement layer and both the colocated macroblocks in the base layer and neighboring macroblocks in the enhancement layer is proposed. The algorithm examines the level of picture details and motion activity, and utilizes the mode information of the base layer to make faster enhancement layer decisions and thus save coding time. Simulation results show that the proposed algorithm reduces encoding by up to 84% compared with the JSVM 9.18 implementation. This is achieved without any noticeable degradation in rate distortion.

In order to improve coding efficiency, a new inter-layer prediction mechanism is incorporated in the SVC extension of H.264/AVC. This utilizes information from the base layer to inform the process of coding the enhancement layer. However this increases the computational requirement. A fast mode decision algorithm that exploits the correlation of a macroblock in the enhancement layer and both the corresponding macroblock in the base layer and neighbouring macroblocks, is proposed. The algorithm also assesses the homogeneity of the picture content and uses the mode information of the base layer to make faster mode selections in the enhancement layer. The fact that larger partition sizes are more suitable for homogeneous regions, and vice versa, is also exploited. Empirical evaluation of the proposed algorithm shows that, for similar rate distortion performance, encoding time is reduced by up to 84% compared with the JSVM9.18 software implementation.

Natural images contain considerable redundancy, some of which is successfully captured using recently developed directional wavelets. In this paper, an affine symmetric image model is considered. It provides a flexible scheme to exploit geometric redundancy. A patch of texture from an image is rotated, scaled and sheared to approximate other similar parts in the image, revealing the self-similarity relation. The general scheme is derived as follows. A texture model is required that identifies structural patterns. Then the affine symmetry is exploited between structural textures at a local level, the objective being to find the minimum residual error by estimating the affine transform relating two patches of texture. Having developed a local model, the methodology is extended to the whole image to estimate the global affine relation. This global model is further developed in a multiresolution framework for multiscale analysis, by which the self similarity of the image is exploited across space and scale. The multiresolution model can be applied to a series of practical problems. Experimental evaluation demonstrates the effectiveness of the approach in affine invariant texture segmentation and image approximation.

A general purpose block-to-block affine transformation estimator is described. The estimator is based on Fourier slice analysis and Fourier spectral alignment. It shows encouraging performance in terms of both speed and accuracy compared to existing methods. The key elements of its success are attributed to the ability to: 1) locate an arbitrary number of affine invariant points in the spectrum that latch onto significant structural features; 2) match the estimated invariant points with the target spectrum by the slicewise phase-correlation; and 3) use affine invariant points to directly compute all linear parameters of the full affine transform by spectral alignment. Experimental results using a wide range of textures are presented. Potential applications include affine invariant image segmentation, registration, affine symmetric image coding, and motion analysis.

A complexity reduction algorithm tailored for the H.264/AVC encoder is described. It aims to alleviate the computational burden imposed by Lagrangian rate distortion optimization in the inter-mode selection process. The proposed algorithm is described as a hierarchical structure comprising three levels. Each level targets different types of macroblocks according to the complexity of the search process. Early termination of mode selection is triggered at any of the levels to avoid a full cycle of Lagrangian examination. The algorithm is evaluated using a wide range of test sequences of different classes. The results demonstrate a reduction in encoding time of at least 40%, regardless of the class of sequence. Despite the reduction in computational complexity, picture quality is maintained at all bit rates.

Many natural textures comprise structural patterns and show strong self-similarity. We use affine symmetry to segment an image into self-similar regions; that is a patch of texture (blocks from a uniformly partitioned image) can be transformed to other similar patches by warping. If the texture image contains multiple regions, we then cluster patches into a number of classes such that the overall warping error is minimized. Discovering the optimal clusters is not trivial and known methods are computationally intensive due to the affine transformation. We demonstrate efficient segmentation of structural textures without affine computation. The algorithm uses Fourier Slice Analysis to obtain a spectral contour signature. Experimental evaluation on structural textures shows encouraging results and application on natural images demonstrates identification of texture objects.

Directional information is an important component of both natural and synthetic images, and it is exploited in many image processing applications. Directional basis analysis is used to capture significant structural information. This paper presents an empirical study of image denoising with directional bases. We consider two distinct approaches. One involves the multi-resolution Fourier transform (MFT) facilitated with a multi-directional selective filter. The other is based on statistics, independent component analysis (ICA) that adaptively decomposes an image into a set of directional bases. We then present a combined approach that benefits from the computational efficiency of the MFT and the data adaptiveness of ICA. Experimental results are compared with those from other recent directional transforms such as the curvelet and directional cosine transform.

A fast algorithm for mesh-based motion estimation employing uniform triangular patches is proposed. The technique utilises an embedded block model to estimate the motion of the mesh grid points. Without the need for time-consuming evaluation, the algorithm reduces the number of search iterations according to the inherent motion. A block-wise coding approach is taken for the motion information, permitting any picture degradation caused by the fast algorithm to be successfully compensated by the residue coding. Simulations on three classes of test sequence show that the proposed algorithm results in a better PSNR-rate performance than the hexagonal matching algorithm. Moreover, a reduction of up to 91% in mesh iterations is obtained

An efficient algorithm for inter-frame coding in the H.264/AVC standard is extended to provide more significant speedup in computational performance for sequences containing high spatial correlation and motion. The proposed scheme features a more sophisticated search process and robust predictions to achieve better PSNR-rate performance for a large range of compression levels. Extensive simulation results demonstrate speedups of between 41% and 68%, with no noticeable deterioration in picture quality or compression ratio, even for the coding of complex video sequences.

A fast inter-mode selection algorithm is proposed to improve the encoder efficiency of the H.264/MPEG-4 AVC standard, but with insignificant degradation in picture quality. The modified fast inter-mode selection (MFInterms) algorithm extends previous work to provide a. more efficient prediction of mode decision. The strategy incorporates temporal similarity detection and the detection of different moving features within a macroblock. Simulation results demonstrate a speed up in encoding time of up to 73% compared with the H.264 benchmark.

Relevance feedback visualisation (RFV) is a technique developed to visualise the feature values of returned results in a content-based image retrieval system that incorporates relevance feedback. RFV is used also to re-sort retrieved results according to user requirements, enable the interactive investigation of pertinent features and permit the discovery of otherwise unidentifiable trends in the dataset. When large numbers of features are involved, manually determining which feature attribute graphs are the most important can be a burdensome task. In this paper, a method for automatically sorting attribute graphs according to their significance in the search operation is introduced. The result is that features worthy of further investigation are immediately identified, the user interface is improved, and the CBIR system is made more effective.

This paper reports two efficient quadtree-based algorithms for variable-size block matching (VSBM) motion estimation. The schemes allow the dimensions of blocks to adapt to local activity within the image, and the total number of blocks in any frame can be varied while still accurately representing true motion. This permits adaptive bit allocation between the representation of displacement and residual data, and also the variation of the overall bit-rate on a frame-by-frame basis. The first algorithm computes the optimal selection of variable-sized blocks to provide the best-achievable prediction error under the fixed number of blocks for a quadtree-based VSBM technique. The algorithm employs an efficient dynamic programming technique utilizing the special structure of a quadtree. Although this algorithm is computationally intensive, it does provide a yardstick by which the performance of other more practical VSBM techniques can be measured. The second algorithm adopts a heuristic way to select variable-sized square blocks. It relies more on local motion information than on global error optimization. Experiments suggest that the effective use of local information contributes to minimizing the overall error. The result is a more computationally efficient VSBM technique than the optimal algorithm, but with a comparable prediction error.

A distributed protocol is proposed for the synchronization of real-time tasks that have variable resource requirements. The protocol is simple to implement and is intended for large-scale distributed or parallel systems in which processes communicate by message passing. Critical sections, even when nested, may be executed on any processor. Thus, given an adequate number of processors, the execution of critical sections can be completely distributed. More significantly, since the protocol enables the distributed allocation of critical sections, the benefits of various allocations can be analyzed and the system optimized to provide minimal blocking. This has important application in global optimization techniques for allocating large numbers of hard real-time tasks in multiprocessor systems.