Diffusion-model-based image super-resolution techniques often face a trade-off between realistic image generation and computational efficiency. This issue is exacerbated when inference times by decreasing sampling steps, resulting in less realistic and hazy images. To overcome this challenge, we introduce a novel diffusion model named PixelBoost that underscores the significance of embracing the stochastic nature of Brownian motion in advancing image super-resolution, resulting in a high degree of realism, particularly focusing on texture and edge definitions. By integrating controlled stochasticity into the training regimen, our proposed model avoids convergence to local optima, effectively capturing and reproducing the inherent uncertainty of image textures and patterns. Our proposed model demonstrates superior objective results in terms of learned perceptual image patch similarity (LPIPS), lightness order error (LOE), peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), as well as visual quality. To determine the edge enhancement, we evaluated the gradient magnitude and pixel value, and our proposed model exhibited a better edge reconstruction capability. Additionally, our model demonstrates adaptive learning capabilities by effectively adjusting to Brownian noise patterns and introduces a sigmoidal noise sequencing method that simplifies training, resulting in faster inference speeds.

Since the emergence of high-quality multimedia processing applications such as video streaming, digital editing, archiving, etc. these days, an intra coding rate control (RC) is becoming an indispensable and important technology. In this paper, a frame-level intra RC scheme for Versatile Video Coding (VVC) using a Lagrange multiplier adjustment (LMA) is proposed. The VVC test model (VTM) uses an R-λ model-based rate control. However, the estimation performance of target bits based on an R-λ-QP relation is decreased because the distortion dependencies among consecutive frames are not considered especially for intra RC. Thus, in a rate-distortion optimization (RDO) based encoding, the λ values determined for given quantization parameter (QP) values should be elaborately controlled to increase the target bits estimation performance. In our work, we focus on the intra RC scheme by taking advantage of particle-filtering-based prediction (PFP) for distortion estimates, and precise per-frame λ values can be derived for an appropriate RDO process that can lead to small bit-fluctuations. Our extensive experimental results demonstrate that our RC scheme using the per-frame LMA is superior to the default RC (VTM-16.0rc1) method and the state-of-the-art RC methods withsignificantmargins of average 15.57%, 15.31% and 31.13% improvements in terms of the normalized root mean square error (NRMSE) for All Intra (AI) configuration of VVC, respectively.

Although recent emotion recognition methods (based on facial expression cues) achieve excellent performance in controlled scenarios, the recognition of emotion in the wild remains a challenging problem because of occlusion, large head poses, illumination variations, etc. Recent advances in deep learning show that combining an ensemble of deep learning models can considerably outperform the approach of using only a single deep learning model for challenging recognition problems. This paper presents a novel ensemble deep learning method, “deep convolutional neural network (DCNN) ensemble classifier”, for improved facial expression recognition (FER) in the wild. Our proposed DCNN ensemble classifier is novel in terms of the following aspects: (1) the process of finding ensemble weights for combining DCNN decision outputs is formulated as a stochastic optimization problem (via simulated annealing) in which the energy to be minimized represents the generalized (test) classification error of the DCNN ensemble and (2) for the creation of DCNN ensemble members, we propose the combined use of different types of face representations and bagging (T. G. Dietterich, 2000), which is quite useful in increasing the diversity of the DCNN ensemble. Extensive and comparative experiments on three wild FER datasets, namely FER2013, SFEW2.0, and RAF-DB, show that the proposed DCNN ensemble classifier achieves competitive FER performances when compared with other recently developed methods—76.69%, 58.68%, and 87.13% of FER accuracy under the FER2013, SFEW2.0, and RAF-DB evaluation protocols, respectively.

In this paper, we propose a new R/D estimation method for intra-predictive coding with deep block partitioning structures. In our proposed R/D prediction, we adopt a particle filtering based prediction (PFP) to precisely predict intermediate R/D estimates for the next frame in a stochastic manner, which helps increasing the prediction accuracy of fast changing R/D values. Then, based on the intermediate R/D estimates by PFP, we infer an optimal model parameter of the TC's probability density function (pdf) via convex optimization. We found that the proposed method brings about more stable R/D estimation performance thanks to both the improved prediction accuracy using the PFP for abrupt changes in true R/D values and the precise estimation of the optimal model parameter. Experimental results show that our method significantly reduces the normalized root mean square error from average 3.17 to 0.79 (74.90% reduction) for rate and from average 2.32 to 0.82 (64.61% reduction) for distortion, compared to the state-of-the art method.

This letter proposes a novel way of using an ensemble of multiple deep convolutional neural networks (DCNNs) for Alzheimer's disease classification, based on magnetic resonance imaging (MRI) images. To create this ensemble of DCNNs, we propose to combine the use of multiple MRI projections (as input) with that of different DCNN architectures to increase the deep ensemble diversity. In particular, to find the optimal fusion weights of the DCNN members, we designed a novel deep ensemble generalization loss, which accounts for interaction and cooperation during the optimal weight search. The optimization framework, equipped with our ensemble generalization loss, was formulated and solved using the sequential quadratic programming. Through this method, we achieved optimal DCNN fusion weights (i.e., a high generalization performance). The experimental results showed that our proposed DCNN ensemble outperforms current deep learning-based methods: it is able to produce state-of-the-art results on the Alzheimer's disease neuroimaging initiative (ADNI) dataset.

Most DCNN-based FR approaches typically employ grayscale or RGB color images as input representations of DCNN architectures. However, other effective face representation methods have been developed and incorporated into current practical FR systems. In light of this fact, the focus of our study is to employ Gabor face representations in the design of DCNN-based FR frameworks to improve FR performance. To this end, we develop a novel "Gabor DCNN (GDCNN)" ensemble method that effectively applies different and multiple Gabor face representations as inputs during the training and testing phases of a DCNN for FR applications. The proposed GDCNN ensemble method primarily consists of two parts: 1) GDCNN ensemble construction and 2) GDCNN ensemble combination. The goal of the former part is to build an ensemble of GDCNN members (i.e., base models), each learned with a particular type of Gabor face representation. The objective of the latter part is to adaptively combine multiple FR outputs of individual GDCNN members. We perform extensive experiments to evaluate our proposed method on four public face databases (DBs) using the associated standard evaluation protocols. Experimental results demonstrate that our approach exhibits significantly better FR performance than typical DCNN-based approaches that rely only on grayscale or color face images as input representations. In addition, the feasibility of our proposed GDCNN ensemble has been successfully demonstrated by making comparisons with other state-of-the-art DCNN-based FR methods.

A video record plays a crucial role in providing evidence for crime scenes or road accidents. However, the main problem with the video record is that it is often vulnerable to various video tampering attacks. Although visual evidence is required to conduct an integrity verification before investigations, it is still difficult for human vision to detect a forgery. In this paper, we propose a novel video integrity verification method (IVM) that takes advantage of a blockchain framework. The proposed method employs an effective blockchain model in centralized video data, by combining a hash-based message authentication code and elliptic curve cryptography to verify the integrity of a video. In our method, video content with a predetermined size (segments) is key-hashed in a real-time manner and stored in a chronologically chained fashion, thus establishing an irrefutable database. The verification process applies the same procedure to the video segment and generates a hash value that can be compared with the hash in the blockchain. The proposed IVM is implemented on a PC environment, as well as on an accident data recorder-embedded system for verification. The experimental results show that the proposed method has better detection capabilities and robustness toward various kinds of tampering, such as copy–move, insert, and delete, as compared to other state-of-the-art methods. An analysis based on execution time along with an increase in the number of blocks within the blockchain shows a minimal overhead in the proposed method.

Joint exploration model (JEM) reference codecs of ISO/IEC and ITU-T utilize multiple types of integer transforms based on DCT and DST of various transform sizes for intra- and inter-predictive coding, which has brought a significant improvement in coding efficiency. JEM adopts three types of integer DCTs (DCT-II, DCT-V, and DCT-VIII), and two types of integer DSTs (DST-I and DST-VII). The fast computations of Integer DCT-II and DST-I are well known, but few studies have been performed for the other types such as DCT-V, DCT-VIII, and DST-VII for all transform sizes. In this paper, we present fast computation methods of N-point DCT-V and DCT-VIII. For this, we first decompose the DCT-VIII into a preprocessing matrix, the DST-VII and a post-processing matrix to quickly compute it by using the linear relation between DCT-VIII and DST-VII. Then, we approximate integer kernels of N = 4, 8, 16, and 32 for DCT-V, DCT-VIII, and DST-VII with norm scaling and bit-shift to be compatible with quantization in each stage of multiplications between decomposed matrices for video coding. In various experiments, the proposed fast computation methods have shown to effectively reduce the total complexity of the matrix operations with little loss in BDBR performance. In particular, our methods reduce the number of addition and multiplication operations by 38% and 80.3%, respectively, in average, compared to the original JEM.

In this paper, we propose a new edge model for edge adaptive graph-based transforms (EA-GBTs) in video compression. In particular, we consider step and ramp edge models to design graphs used for defining transforms, and compare their performance on coding intra and inter predicted residual blocks. In order to reduce the signaling overhead of block-adaptive coding, a new edge coding method is introduced for the ramp model. Our experimental results show that the proposed methods outperform classical DCT-based encoding and that ramp edge models provide better performance than step edge models for intra predicted residuals.

This paper introduces a novel class of transforms, called graph-based separable transforms (GBSTs), based on two line graphs with optimized weights. For the optimal GBST construction, we formulate a graph learning problem to design two separate line graphs using row-wise and column-wise residual block statistics, respectively. We also analyze the optimality of resulting separable transforms for both intra and inter predicted residual block models. Moreover, we show that separable DCT and ADST (DST-7) are special cases of the GBSTs. Our experimental results demonstrate that the proposed optimized transforms outperform 2-D DCT/ADST and separable KLT.

In this paper, a low complexity coding unit (CU)-level rate and distortion estimation scheme is proposed for High Efficiency Video Coding (HEVC) hardware-friendly implementation where a Walsh-Hadamard transform (WHT)-based low-complexity integer discrete cosine transform (DCT) is employed for distortion estimation. Since HEVC adopts quadtree structures of coding blocks with hierarchical coding depths, it becomes more difficult to estimate accurate rate and distortion values without actually performing transform, quantization, inverse transform, de-quantization, and entropy coding. Furthermore, DCT for rate-distortion optimization (RDO) is computationally high, because it requires a number of multiplication and addition operations for various transform block sizes of 4-, 8-, 16-, and 32-orders and requires recursive computations to decide the optimal depths of CU or transform unit. Therefore, full RDO-based encoding is highly complex, especially for low-power implementation of HEVC encoders. In this paper, a rate and distortion estimation scheme is proposed in CU levels based on a low-complexity integer DCT that can be computed in terms of WHT whose coefficients are produced in prediction stages. For rate and distortion estimation in CU levels, two orthogonal matrices of 4×4 and 8×8 , which are applied to WHT that are newly designed in a butterfly structure only with addition and shift operations. By applying the integer DCT based on the WHT and newly designed transforms in each CU block, the texture rate can precisely be estimated after quantization using the number of non-zero quantized coefficients and the distortion can also be precisely estimated in transform domain without de-quantization and inverse transform required. In addition, a non-texture rate estimation is proposed by using a pseudoentropy code to obtain accurate total rate estimates. The proposed rate and the distortion estimation scheme can effectively be used for HW-friendly implementation of HEVC encoders with 9.8% loss over HEVC full RDO, which much less than 20.3% and 30.2% loss of a conventional approach and Hadamard-only scheme, respectively.

In this paper, an all-zero block detection scheme is proposed prior to DCT to reduce the encoding complexity for high efficiency video coding (HEVC). Since many coding blocks tend to have all zero coefficients after DCT and quantization, it is worthwhile to detect all-zero-quantized blocks for input residual blocks before DCT so that subsequent transform and quantization can be skipped. Unlike previous coding standards, HEVC adopts large transform sizes such as 16 × 16 and 8 × 8. It becomes more difficult to accurately detect all-zero blocks in HEVC because the large transform blocks contains more variety of content characteristics than smaller ones, thus making it ineffective the existing all-zero block (AZB) detection schemes for large transform blocks in HEVC. In this paper, a novel AZB detection scheme is proposed for the case that Hadamard transform is used as a distortion metric for RDO in HEVC. Statistical upper bounds to be all-zero blocks are derived using the relationship between Walsh Hadamard and DCT transform kernels. Then, a small number of quantized coefficients in a upper left corner of a transform block, which are obtained using the relations between Hadamard transform and DCT, are examined for AZB detection. For 32 × 32 blocks, DC coefficients of 8 × 8 sub-blocks are further examined for AZB detection. The experimental results demonstrate that the proposed scheme detects 87.79% of actual AZBs with 2.87% false alarm rate in average, outperforming the state-of-the-art method. Computational complexity to detect AZB is almost negligible compared to the conventional method.

Recently, a new video coding standard, High Efficiency Video Coding (HEVC), has shown greatly improved coding efficiency by adopting hierarchical structures of coding unit (CU), prediction unit (PU), and transform unit (TU). To best achieve the coding efficiency, the best combinations of CU, PU, and TU must be found in the sense of the minimum rate-distortion (R-D) costs. Owing to this, a large computational complexity occurs. Among these CU, PU, and TU, the determination of CU sizes most significantly affects the R-D performance of HEVC encoders, which causes large computational costs in operation with PU and TU size determinations. In spite of recent works in the complexity reduction of HEVC encoders, most of the research has focused on the complexity reduction with fast CU split in intra slice coding and with early TU split in both intra and inter slice. In this paper, we propose a fast and an efficient CU encoding scheme based on the spatiotemporal encoding parameters of HEVC encoders, which consists of an improved early CU SKIP detection method and a fast CU split decision method. For the current CU block under encoding, the proposed scheme utilizes sample-adaptive-offset parameters as the spatial encoding parameter to estimate the texture complexity that affects the CU partition. In addition, the motion vectors, TU size, and coded block flag information are used as the temporal encoding parameters to estimate the temporal complexity that also affects the CU partition. The proposed scheme effectively utilizes the spatiotemporal encoding parameters that are the byproducts during the encoding process of HEVC without additionally required computation. The proposed novel fast CU encoding scheme significantly reduces the total encoding time with negligible RD-performance loss. The experimental results show that the proposed scheme achieves the total encoding time savings of average 49.6% and 42.7% only with average 1.4% and 1.0% bit-rate losses for various test sequences under random access and low delay B conditions, respectively. The proposed scheme has an advantage on the implementation for parallel processing in pipeline structures of HEVC encoders due to its independency with neighboring CU blocks.

In this study, the performance of hierarchical transform coding is analysed with design of an order‐16 integer transform kernel. The proposed hierarchical transform‐coding structure is constructed with a set of 4 × 4, 8 × 8 and 16 × 16 integer transforms of variable transform block sizes, which takes the advantages of both lower and higher transform kernels by flexibly adapting to varying image characteristics of video sequences with homogeneous and complex regions. The proposed hierarchical transform‐coding structure is implemented as an extension to H.264/advanced video coding joint model. The authors show the effectiveness of the hierarchical variable‐sized block transform scheme by analysing the quantisation effects and the correlation among neighbouring pixels in video sequences of different spatial resolutions. The experimental results show that: (i) the variable‐sized block transform scheme with the hierarchical structure is advantageous to the texture regions with strong local edges and (ii) the higher‐order‐16 integer transform kernel itself is more effective for the homogeneous texture regions, which are often encountered in higher resolution sequences. Therefore these two features can complementarily work in an rate‐distortion (RD) optimised manner for various characteristics of the input signals.

In this paper, a frame-level rate control scheme is proposed based on texture and nontexture rate models for High Efficiency Video Coding (HEVC). Due to more complicated coding structures and the adoption of new coding tools, the statistical characteristics of transform residues are significantly different depending on the depth levels of coding units (CUs) from which the residues are obtained. A new texture rate model is constructed for the transform residues, which are categorized into three types of CUs: low-, medium- and high-textured CUs. One single Laplacian probability PDF model is used for each residue category to derive a rate-quantization model. Based on the Laplacian PDF, a simplified rate model for texture bits is derived using entropy. In addition, an analytic rate model for nontexture bits is proposed, which also takes into account the different characteristics of nontexture bits occurring in various depths of CUs in HEVC. The nontexture bitrates are modeled based on the linear relation between the total nontexture data and the dominant nontexture data in each CU category. Based on the proposed rate models for the texture and nontexture bits, accurate rate control can be achieved owing to more precise rate estimation. The experimental results show that the proposed rate control scheme achieves the average PSNR with 0.44 dB higher and the average PSNR standard deviation of 0.32 point lower with the buffer status levels maintained very close to target buffer levels, compared to the conventional methods. Finally, the proposed rate control scheme remarkably outperforms the conventional schemes especially for the sequences of complex texture and large motion.

Inprobability model based rate control of video coding, modeling of residual distribution is important in predicting precise distortions so as to determine appropriate quantization parameter values. For this, single probability model approaches have been popularly taken which may fail to model the underlying statistical characteristics of different residues from variable block-sized coding. In this letter, new rate and distortion models based on a mixture of multiple Laplacian distributions are presented for the transform coefficients of inter-predicted residues in quadtree coding. The proposed mixture model of multiple Laplacian distributions is tested for the High Efficiency Video Coding (HEVC) Test Model (HM) with quadtree-structured Coding Unit and Transform Unit. The experimental results show that the proposed model achieves more accurate results of rate and distortion estimation than the single probability models.

In this paper, a fast mode decision method for the spatial higher layers (SHLs) in scalable video coding (SVC) is proposed based on coding dependency between two adjacent spatial lower and higher layers. The proposed fast mode decision method detects zero motion and zero transform coefficient blocks in the current spatial layer using the already encoded information of the corresponding blocks from the lower layer. The information for zero motion vectors and zero transform coefficients is used to induce a reduced set of candidate modes in SHLs of SVC, which reduces the computation complexity up to about 75% of the total encoding time while maintaining the coding performance with negligible amounts of degradation in peak signal-to-noise ratio values and bitrates.

In this paper, a rate-distortion optimized variable block transform coding scheme is proposed based on a hierarchical structured transform for macroblock (MB) coding with a set of the order-4 and −8 integer cosine transform (ICT) kernels of H.264/AVC as well as a new order-16 ICT kernel. The set of order-4, −8 and −16 ICT kernels are applied for inter-predictive coding in square (4×4, 8×8 or 16×16) or non-square (16×8 or 8×16) transform for each MB in a hierarchical structured manner. The proposed hierarchical variable-sized block transform scheme using the order-16 ICT kernel achieves significant bitrate reduction up to 15%, compared to the High profile of H.264/AVC. Even if the number of candidates for the transform types increases, the encoding time can be reduced to average 4–6% over the H.264/AVC