Neural Network-Based Intra Prediction (NN-Intra) predicts a block using its reference samples with a neural network. It has been actively studied in Neural Network-Based Video Coding (NNVC) by JVET and was recently adopted in the Enhanced Compression Model (ECM) considering its high prediction performance. Despite efforts to reduce complexity of the NN-Intra made during its adaptation process from NNVC to ECM, its complexity still remains high, and more investigation is needed on the transform process after prediction. This paper explores a method to transform the NN-Intra-coded block using DCT-based transform sets in ECM. In this regard, we merge transform sets for directional and non-directional modes to generate a merged transform set from which a transform kernel pair is selected for the block. Experimental results ECM-15.0 demonstrate that our method achieves a coding gain of 0.01% in luma channel and has encoder and decoder complexity of 100.2% and 99.8%, respectively.

Thanks to plenoptic cameras, rich information on the radiance of a scene can be conveniently captured without heavy devices like camera arrays. However, rendering techniques are needed to generate views for human visual perception. Existing patch extraction-based rendering techniques can generate views from the lenslet images captured by plenoptic cameras, but they suffer from the inherent problem of artifacts and the limited views to be rendered. In this paper, we present a new view rendering technique from plenoptic 2.0 camera-captured lenslet image by using 3-dimensional gaussian splatting (3DGS). At its first step, the reference lenslet converter (RLC) provided by MPEG LVC AhG, one of the existing patch extraction methods, generates initial views with the help of estimated disparity between adjacent micro images in the lenslet image. At its second step, the 3DGS generates the final views after being trained by the initial views. The rendering results obtained by the proposed 2-step approach show significantly fewer artifacts in the rendered views than the patch stitching process of the existing method.

This paper introduces a novel intra-prediction scheme for coding plenoptic video which can effectively exploit large correlation between current and neighboring macropixel images. While the intra block copy method is well recognized as a promising coding tool for plenoptic video, it has fundamental issues like much searching time for block vectors (BVs) and more bits to encode these BVs into a bitstream. Our method can effectively solve them by pre-defining the prediction candidates to save encoding time and signaling only the index of prediction location instead of BVs to reduce the overhead bits for encoding BVs. Compared to HEVC, our method is experimentally shown to achieve an average bitrate gain of about $19.70 \%$ and $11.99 \%$ respectively under the AI-Main and RA-Main conditions. Moreover, better trade-off can be made between complexity and coding performance than existing methods.

Various deep learning-based methods have shown excellent performances in converting images from Low Dynamic Range (LDR) to High Dynamic Range (HDR). Many of them employ architectures like Autoencoders or U-Nets, and demonstrate significant improvements in performance, however, they demand large network sizes and associated computational loads. It leads to serious issues of overheating and power consumption, especially for average users handling many LDR images on personal devices such as smartphones, laptops, and tablets. Our study addresses this issue using reinforcement learning aiming for a practical solution. In this approach, we integrate a lightweight agent with traditional methods by designing a simple and effective reward function to ensure that the lightweight agent could effectively execute the traditional methods. Consequently, we perform the task of HDR conversion using a network that requires relatively low computational resources.

In this paper we introduce a novel approach to better utilize the intra block copy (IBC) prediction tool in encoding lenslet light field video (LFV) captured using plenoptic 2.0 cameras. Although the IBC tool has been recognized as promising for encoding LFV content, its fundamental limit due to its original design rooted for encoding conventional videos suggests slight modification possibility to better suit the property of LFV content. Observing the inherently large amount of repetitive image patterns due to the microlens array (MLA) structure of plenoptic cameras, several techniques are suggested in this paper to enhance the IBC coding tool itself for more efficiently encoding LFV contents. Our experimental results demonstrate that the proposed method significantly enhances the IBC coding performance in case of encoding LFV contents while concurrently reducing encoding time.

Being a general representation format of the dense light field, lenslet video, where each frame consists of a 2D grid of micro-images, shows high potential for applications in immersive media such as glasses-free 3D displays and virtual reality. However, its distinct spatial-temporal-angular distribution places a significant challenge on conventional video coding. In July 2021, the Moving Picture Experts Group (MPEG) established an Ad-Hoc group, Lenslet Video Coding (LVC), to explore use cases, efficient compression methods, testing sequences, conversion tools, and coding architectures towards a new compression standard. This paper provides an overview of recent progress in the LVC Ad Hoc group and presents the compression efficiency of state-of-the-art codec agnostic coding tools to encourage contributions in the future.

Light field (LF) rescaling is indispensable in accommodating different LF image resolutions for different applications. Unlikely most recent studies which only execute learned LF upscaling from a predefined downscaling method, we propose a novel LF rescaling framework by jointly optimizing learned LF downscaling and upscaling as a combined task. Specifically, our light field rescaling network (LFRN) simultaneously extracts features from different 2D subspaces of LF data (e.g., spatial-angular and epipolar subspaces) to fully handle 4D LF image information. Our newly designed attention fusion module (AFM) adaptively combines these two data features based on learnable embedding weights. Due to joint optimization of the learned LF downscaling and upscaling tasks, our LFRN method can achieve significant performance gain in both objective and subjective visual qualities compared to conventional predefined downscaling with learned LF upscaling task.

The rate-distortion optimized quantization (RDOQ) used in video encoding helps to achieve high compression performance but leads to huge computation. We experimentally observe that in approximately half of the quantization blocks, RDOQ does not change the quantization results initially obtained by the conventional scalar quantizer. In this context, we design a machine learning-based quantizer selection model which lets an encoder decide whether or not to apply RDOQ process for a given transform block (TB) in advance. Our experiments show that the proposed complexity-efficient quantizer selection model reduces 9% and 35% respectively of the encoding and quantization time with BDBR loss of only 0.03%. The proposed selective quantizer achieves almost the same coding performance of RDOQ applied all the time with only around 20% of its actual usage.

This paper proposes a novel hybrid light field (LF) denoising method which is based on a convolutional neural network (CNN) designed to reflect the characteristic of LF image in both pixel and frequency domains. Noting that the image noise usually has much high-frequency energy, the proposed network is designed to operate in a transform domain in two stages. At the first stage, energy compaction of spatial-angular information of LF image is sought by 4D-DCT separated transform which can achieve better energy compaction than 2D-DCT applied separately in the spatial and angular domain. The transformed LF is decomposed into different frequency components and each frequency component is recovered progressively. Subsequently, we reshape and convert different frequency components into pixel domain to perform the next refinement step for which a residual spatial-angular block (RSAB) is proposed to handle the 4D LF structure in the pixel domain. Extensive experimental results on different noisy datasets confirm the effectiveness of our proposed method compared to state-of-the-art methods in both objective and subjective quality.

Multi-focused plenoptic images possess many special characteristics related to the micro-images (MIs) array, which are expected to be useful in further increasing its compression performance. Those special characteristics come from the much overlap and sharpness variance among its micro-images, and proper handling of such properties can lead to better patch-based prediction. In this paper, for multi-focused plenoptic image data, we design a new prediction model taking into account the disparity shift constraint coming from the overlaps and the sharpness variation. Experiment results show coding gain respectively of 21% over the HEVC Intra and 27% when the proposed method is combined with the Intra Block Copy (IBC) tool which is reported very effective in plenoptic image coding.

One of the design goals of the recently published international video coding standard, Versatile Video Coding (VVC/H.266), is efficient coding of computer-generated video content (commonly referred to as screen content) which exhibits different signal characteristics from the usual camera-captured video (commonly referred as natural content). VVC can perform transform in multiple different ways including skipping the transform itself, which demands much computation for its best selection among many combinatory options. In this paper, we investigate designing a machine-learning-based early transform skip mode decision (ML-TSM) which makes a determination whether or not to skip the transform in an early stage by making a simple classification employing key features designed in such a way to reflect the characteristics of TSM blocks well. Compared with the VVC reference software 14.0, the proposed scheme is verified to reduce computational complexity by 11% and 4% with a Bjøntegaard delta bitrate (BDBR) increase of 0.34% and 0.23% respectively under all-intra (AI) and random-access (RA) configurations.

Despite the remarkable progress made by the salient object detection of natural sensing images (NSI-SOD), the complex background and scale diversity issues of remote sensing images (RSIs) still pose a substantial obstacle. In this study, we build an end-to-end channel-enhanced remodeling-based network (CRNet) for optical RSIs (ORSIs) to highlight salient objects through feature augmentation. First, the backbone convolutional block is used to suggest the fundamental characteristics. Then, we use the channel enhance module (CEM) to enhance the shallow features. CEM primarily relies on the channel attention mechanism and employs a no-downscaling strategy to produce local cross-channel interaction, which lowers model complexity while enhancing extraction performance. Meanwhile, we use the redefined feature module (RFM) to reconstruct the deep features and generate global attention features by dimensional transformation and feature relationship aggregation to achieve the role of locating salient targets. Finally, the cascade combines the multi-scale features to provide the final saliency map. To further enhance the representational power of the network, we use a hybrid loss function to improve performance. The proposed approach outperforms current state-of-the-art methods, as shown by several experiments on three available datasets. The source code of the proposed CRNet is available publicly at https://github.com/hilitteq/CRNet.git.

The joint use of multisource remote-sensing (RS) data for Earth observation missions has drawn much attention. Although the fusion of several data sources can improve the accuracy of land-cover identification, many technical obstacles, such as disparate data structures, irrelevant physical characteristics, and a lack of training data, exist. In this article, a novel dual-branch method, consisting of a hierarchical convolutional neural network (CNN) and a transformer network, is proposed for fusing multisource heterogeneous information and improving joint classification performance. First, by combining the CNN with a transformer, the proposed dual-branch network can significantly capture and learn spectral–spatial features from hyperspectral image (HSI) data and elevation features from light detection and ranging (LiDAR) data. Then, to fuse these two sets of data features, a cross-token attention (CTA) fusion encoder is designed in a specialty. The well-designed deep hierarchical architecture takes full advantage of the powerful spatial context information extraction ability of the CNN and the strong long-range dependency modeling ability of the transformer network based on the self-attention (SA) mechanism. Four standard datasets are used in experiments to verify the effectiveness of the approach. The experimental results reveal that the proposed framework can perform noticeably better than state-of-the-art methods. The source code of the proposed method will be available publicly athttps://github.com/zgr6010/Fusion_HCT.git.

Plenoptic images and videos bearing rich information demand a tremendous amount of data storage and high transmission cost. While there has been much study on plenoptic image coding, investigations into plenoptic video coding have been very limited. We investigate the motion compensation (or so-called temporal prediction) for plenoptic video coding from a slightly different perspective by looking at the problem in the ray-space domain instead of in the conventional pixel domain. Here, we develop a novel motion compensation scheme for lenslet video under two sub-cases of ray-space motion, that is, integer ray-space motion and fractional ray-space motion. The proposed new scheme of light field motion-compensated prediction is designed such that it can be easily integrated into well-known video coding techniques such as HEVC. Experimental results compared to relevant existing methods have shown remarkable compression efficiency with an average gain of 20.03% and 21.76% respectively under "Low delayed B " and "Random Access" configurations of HEVC.

Low-rank tensor representation philosophy has enjoyed a reputation in many hyperspectral image (HSI) low-level vision applications, but previous studies often failed to comprehensively exploit the low-rank nature of HSI along different modes in low-dimensional subspace, and unsurprisingly handled only one specific task. To address these challenges, in this paper, we figured out that in addition to the spatial correlation, the spectral dependency of HSI also implicitly exists in the coefficient tensor of its subspace, this crucial dependency that was not fully utilized by previous studies yet can be effectively exploited in a cascaded manner. This led us to propose a unified subspace low-rank learning regime with a new tensor cascaded rank minimization, named STCR, to fully couple the low-rankness of HSI in different domains for various low-level vision tasks. Technically, the high-dimensional HSI was first projected into a low-dimensional tensor subspace, then a novel tensor low-cascaded-rank decomposition was designed to collapse the constructed tensor into three core tensors in succession to more thoroughly exploit the correlations in spatial, nonlocal, and spectral modes of the coefficient tensor. Next, difference continuity-regularization was introduced to learn a basis that more closely approximates the HSI's endmembers. The proposed regime realizes a comprehensive delineation of the self-portrait of HSI tensor. Extensive evaluations conducted with dozens of state-of-the-art (SOTA) baselines on eight datasets verified that the proposed regime is highly effective and robust to typical HSI low-level vision tasks, including denoising, compressive sensing reconstruction, inpainting, and destriping. The source code of our method is released at https://github.com/CX-He/STCR.git.

Video stabilization is the process of improving the video quality by removing annoying fluctuant motion caused by camera jittering. A key issue of a successful solution is the temporal adaptability to motion and the overall robustness with respect to different motion types. However, most previous methods usually produce non-motion adaptive stabilized videos. In other words, under-smoothing in slow motion segments and over-smoothing in rapid motion segments will be produced for complex shaky videos. To overcome these drawbacks, we propose a novel video stabilization approach using a motion morphological component (MMC) decomposition. Specifically, the observed motion is decomposed into three MMCs: low-frequency smoothed (LFS) motion, high-frequency compensatory (HFC) motion, and shaky motion. LFS motion helps to largely stabilize videos, and HFC motion helps to recover missing motion to deal with over-smoothing. Subsequently, we present an MMC-based model to retrieve the desired smoothed motion, in which weighted nuclear norm and autoregression priors are used for LFS motion, while a sparsity prior is adopted for HFC motion. In addition, we design an adaptive weight setting scheme to detect rapid motions and to calculate the optimal weights. Finally, we develop a stabilization algorithm under the Alternating Direction Method of Multipliers (ADMM) framework. Experimental results demonstrate that our method can achieve high-quality results compared with that of other state-of-the-art stabilization methods in terms of robustness and efficiency, both quantitatively and qualitatively.

In raw plenoptic video, the optimal motion searching points mostly follow the hexagonal structure of micro-images. Based on this understanding, we propose a new motion vector resolution, namely the hexagonal lattice (HL) resolution which reflects micro-image structure. The HL resolution can be efficiently represented by HL basis. A study in this paper shows that motion vectors are highly concentrated at hexagonal lattice points, leading to use of the proposed resolution in the context of video compression. In this regard, we demonstrate the compression benefit brought by estimating motion vectors at HL resolution in the VVC codec.

This paper proposes a new downsampling-based light field video coding (D-LFVC) framework whose success relies on how to design an effective restoration method that can remove artifacts brought by both downsampling and compression. Since light field (LF) video is of high dimensionality data, the restoration methods designed for conventional 2D video are sub-optimal solutions for our D-LFVC. In this regard, we design a new restoration network, named "LF-QEN," for our D-LFVC framework. Specifically, the network contains three different feature extractor modules, allowing us to simultaneously exploit information from different kinds of 4D LF representation: spatial, angular, and epipolar image information. Our experimental results show that, compared to compression by HEVC-SCC standard, the proposed framework can obtain not only nearly 50% bitrate savings but also can significantly enhance the quality of decoded LF video.

Correlation filter (CF) has drawn extensive interest in aerial object tracking due to its remarkable performance. Recently, the popular CF methods based on temporal–spatial regularization have been proved to be able to effectively improve the tracking results. However, the boundary effect and filter template degradation still influence the speed and accuracy of the trackers. To handle the two problems, a novel dynamic temporal–spatial regularization-based channel weighted tracking (DTSCT) method was proposed in this work. First, we attempted to employ the saliency detection technique to describe object variation for weakening the boundary effect. Then, the filter template was introduced to the temporal regularization to alleviate the template degradation. In addition, an adaptive weighting strategy was utilized to remove data redundancy in the feature channels. Experiments on three benchmark datasets showed the competitive performance of our DTSCT approach compared to the state-of-the-art methods.

Hyperspectral image (HSI) classification refers to accurately corresponding each pixel in an HSI to a land-cover label. Recently, the successful application of multiscale and multifeature methods has greatly improved the performance of HSI classification due to their enhanced utilization of the available spectral–spatial information. However, as the number of scales and the number of features increases, it becomes more difficult to achieve an optimal degree of fusion for multiple classifiers [e.g., kernel extreme learning machine (KELM)]. On the other hand, a limited sample size of the HSI may cause overfitting problems, which seriously affects the classification accuracy. Therefore, in this article, a novel multi-structure KELM with attention fusion strategy (MSAF-KELM) is proposed to achieve accurate fusion of multiple classifiers for effective HSI classification with ultrasmall sample rates. First, a multi-structure network is built, which combines multiple scales and multiple features to extract abundant spectral–spatial information. Second, a fast and efficient KELM is employed to enable rapid classification. Finally, a weighted self-attention fusion strategy (WSAFS) is introduced, which combines the output weights of each KELM subbranch and the self-attention mechanism to achieve an efficient fusion result on multi-structure networks. We conducted experiments on four types of HSI datasets with different evaluation methods and compared them with several classical and state-of-the-art methods, which demonstrate the excellent performance of our method on ultrasmall sample rates. The code is available athttps://github.com/Fang666666/MSAF-KELMfor reproducibility.

Content-based Image Retrieval (CBIR) techniques have been extensively studied with the rapid growth of digital images. Generally, CBIR service is quite expensive in computational and storage resources. Thus, it is a good choice to outsource CBIR service to the cloud server that is equipped with enormous resources. However, the privacy protection becomes a big problem, as the cloud server cannot be fully trusted. In this paper, we propose an outsourced CBIR scheme based on a novel bag-of-encrypted-words (BOEW) model. The image is encrypted by color value substitution, block permutation, and intra-block pixel permutation. Then, the local histograms are calculated from the encrypted image blocks by the cloud server. All the local histograms are clustered together, and the cluster centers are used as the encrypted visual words. In this way, the bag-of-encrypted-words (BOEW) model is built to represent each image by a feature vector, i.e., a normalized histogram of the encrypted visual words. The similarity between images can be directly measured by the Manhattan distance between feature vectors on the cloud server side. Experimental results and security analysis on the proposed scheme demonstrate its search accuracy and security.

In deep convolutional neural networks (DCNNs) for single image super-resolution (SISR), the dense and residual feature refinement helps to stabilize the training network and enriches the feature values. However, most SISR networks do not fully exploit the rich feature information in the hierarchical dense residual connections, thus achieving relatively low performance. Besides, in many cases, a large model is not feasible to deploy on mobile or embedded devices. By exploiting the hierarchical dense residual learning, this paper proposes a fast and efficient hierarchical dense residual network (HDRN) to solve these problems. Specifically, we develop a dense compact residual group (DCRG), consisting of several compact residual blocks (CRB), which helps to increase the reusable feature capability. Our experimental results confirm that the proposed HDRN achieves better trade-off between the performance and computational costs than those state-of-the-art lightweight SISR methods.

The motion estimation which plays an important role in video coding requires much computation for encoding. In this paper, from the ray motion characteristics in the lenslet plenoptic video, we derive a new motion search model and propose a fast and efficient microlens-based motion search method. Theoretical analysis and experimental results have verified the new model and demonstrated its efficiency in search. Under the HEVC random-access configuration, we achieve not only substantial encoding time reduction (56.7%), but also bitrate saving of 1.3% on average compared to relevant existing works. Under the low delay configuration, the performances are 23.3% and 2.3%, respectively for encoding time reduction and bitrate saving.

In this paper, we design an improved hard-decision quantization (HDQ) scheme for HEVC compression. A decision tree model is generated based on the behavior of the soft-decision quantization (SDQ) in HEVC, and it is utilized to help making decision for each quantized level in the proposed HDQ. Experimental results show that our proposed quantization scheme achieves an average of 3.11% coding gain compared to the conventional HDQ and it provides a more hardware friendly implementation than SDQ.

This paper proposes a novel data cost that combines spatial and spectral entropies to handle the occlusion problem in the light field depth estimation. In previous works, the spatial entropy data cost has been demonstrated to reduce the effect of occluded pixels in an angular patch (i.e., micro-lens pixel) and to yield an accurate depth value in the presence of occlusion. However, our observation notes that the spatial entropy data cost metric is less reliable when the angular resolution becomes smaller as in light field images. In this paper, we propose a new data cost which integrates a proposed spectral entropy data cost with the spatial entropy data cost. An initial depth map which is estimated using the proposed new data cost is further optimized by the standard graph-cut algorithm and filtered by using an edge-preserving filter. Experimental results have confirmed the effectiveness of the proposed method which achieves more accurate depth values even when the angular resolution becomes smaller.

The increasing prevalence of VR/AR as well as the expected availability of Light Field (LF) display soon call for more practical methods to transmit LF image/video for services. In that aspect, the LF video coding should not only consider the compression efficiency but also the view random-access capability (especially in the multi-view-based system). The multi-view coding system heavily exploits view dependencies coming from both inter-view and temporal correlation. While such a system greatly improves the compression efficiency, its view random-access capability can be much reduced due to so called "chain of dependencies." In this paper, we first model the chain of dependencies by a tree, then a cost function is used to assign an importance value to each tree node. By travelling from top to bottom, a node of lesser importance is cut-off, forming a pruned tree to achieve reduction of random-access complexity. Our tree pruning method has shown to reduce about 40% of random-access complexity at the cost of minor compression loss compared to the state-of-the-art methods. Furthermore, it is expected that our method is very lightweight in its realization and also effective on a practical LF video coding system.

Kernel methods, e.g., composite kernels (CKs) and spatial-spectral kernels (SSKs), have been demonstrated to be an effective way to exploit the spatial-spectral information nonlinearly for improving the classification performance of hyperspectral image (HSI). However, these methods are always conducted with square-shaped window or superpixel techniques. Both techniques are likely to misclassify the pixels that lie at the boundaries of class, and thus a small target is always smoothed away. To alleviate these problems, in this paper, we propose a novel patch-based low rank component induced spatial-spectral kernel method, termed LRCISSK, for HSI classification. First, the latent low-rank features of spectra in each cubic patch of HSI are reconstructed by a low rank matrix recovery (LRMR) technique, and then, to further explore more accurate spatial information, they are used to identify a homogeneous neighborhood for the target pixel (i.e., the centroid pixel) adaptively. Finally, the adaptively identified homogenous neighborhood which consists of the latent low-rank spectra is embedded into the spatial-spectral kernel framework. It can easily map the spectra into the nonlinearly complex manifolds and enable a classifier (e.g., support vector machine, SVM) to distinguish them effectively. Experimental results on three real HSI datasets validate that the proposed LRCISSK method can effectively explore the spatial-spectral information and deliver superior performance with at least 1.30% higher OA and 1.03% higher AA on average when compared to other state-of-the-art classifiers.

The raw video data can be compressed much by the latest video coding standard, high efficiency video coding (HEVC). However, the block-based hybrid coding used in HEVC will incur lots of artifacts in compressed videos, the video quality will be severely influenced. To settle this problem, the in-loop filtering is used in HEVC to eliminate artifacts. Inspired by the success of deep learning, we propose an efficient in-loop filtering algorithm based on the enhanced deep convolutional neural networks (EDCNN) for significantly improving the performance of in-loop filtering in HEVC. Firstly, the problems of traditional convolutional neural networks models, including the normalization method, network learning ability, and loss function, are analyzed. Then, based on the statistical analyses, the EDCNN is proposed for efficiently eliminating the artifacts, which adopts three solutions, including a weighted normalization method, a feature information fusion block, and a precise loss function. Finally, the PSNR enhancement, PSNR smoothness, RD performance, subjective test, and computational complexity/GPU memory consumption are employed as the evaluation criteria, and experimental results show that when compared with the filter in HM16.9, the proposed in-loop filtering algorithm achieves an average of 6.45% BDBR reduction and 0.238 dB BDPSNR gains.

Deep learning-based compressive sensing (DCS) has improved the single scale compressive sensing (CS) with fast and high reconstruction quality. Researchers have further extended it to multi-scale DCS which improves reconstruction quality based on Wavelet decomposition. In this work, we mimic the Difference of Gaussian via convolution and propose a scheme named as Difference of Convolution-based multi-scale DCS (DoC-DCS). Unlike the multi-scale DCS based on a well-designed filter in the wavelet domain, our DoC-DCS jointly learns decomposition, sampling, and reconstruction thereby outperforms other state-of-the-art deep learning based CS methods.

In this paper, we propose a novel cross total variation regularized unidirectional nonlocal low rank tensor approximation method for hyperspectral image denoising. It fully explores the spectral-spatial correlation and non-local self-similarity simultaneously in tensor case and points out that the nonlocal self-similarity is the most important for precisely restoring the HSI. Following the research line in [1], we propose to embed the cross total variation (CrTV) regularization into the unidirectional low rank tensor framework to alleviate the common consistency issue of pixels in overlapped regions. CrTV shows great power to explore the spatial-spectral correlation and has great ability to keep the fine spatial details and preserve the spectra in the course of HSI denoising. The final model can be effectively solved by the alternating direction methods of multipliers (ADMM). Experimental results on HSI data sets validate that the complementary priors (i.e., spatial-spectral correlation and non local self-similarity) really contribute to the performance and also illustrate the superiority of the proposed method when compared with other state-of-the-art denoising methods.

This paper proposes a novel mixed noise removal method via subspace low rank representation and BM4D filtering for hyperspectral imagery (HSI). The proposed method is based on the following two facts. The first one is that the spectra in each class of HSI lie in different low-rank subspace, that is, the HSI data could be decomposed into two sub-matrices with lower ranks in the framework of subspace low rank representation. The second one is that the spatial structures of HSI have the property of non-local self-similarity (NSS), and the NSS could be effectively exploited by BM4D filter with no additional parameters. The proposed model can be easily and effectively solved by splitting it into several sub-problems via the alternating direction method of multipliers (ADMM). Experimental results validate that the proposed method outperforms other state-of-the-art denoising methods for HSI.

With joint learning of sampling and recovery, the deep learning-based compressive sensing (DCS) has shown significant improvement in performance and running time reduction. Its reconstructed image, however, losses high-frequency content especially at low subrates. This happens similarly in the multi-scale sampling scheme which also samples more low-frequency components. In this paper, we propose a multi-scale DCS convolutional neural network (MS-DCSNet) in which we convert image signal using multiple scale-based wavelet transform, then capture it through convolution block by block across scales. The initial reconstructed image is directly recovered from multi-scale measurements. Multi-scale wavelet convolution is utilized to enhance the final reconstruction quality. The network is able to learn both multi-scale sampling and multi-scale reconstruction, thus results in better reconstruction quality.

Recently, the hidden Markov model (HMM) with student’s t-mixture model (SMM), called student’s t-HMM (SHMM) for short, has received much attention in unsupervised learning of sequential data. However, the current existing SHMMs fail to take into consideration of the relevant features embedded in local subspaces, thus influencing their performances in clustering. To address the problem, a novel SHMM is proposed by combining the measure of localized feature saliency (LFS) with SMM and utilizing two student’s t-distributions as subcomponents to respectively describe the distributions of useful features and non-salient “features,” with the purpose of accurately modeling the hidden state observation emission distributions of SHMM. Moreover, we exploit the variational Bayesian learning technique to simultaneously estimate the LFS, the number of components and other parameters of the herein proposed SHMM. Experimental results on both synthetic and real data sets demonstrate the improved robustness, effectiveness, and accuracy of our model.

In this paper, a new classifier under Bayesian framework is proposed to explore homogeneous region based low rank representation in hidden field for classification of hyperspectral imagery (HSI). This classifier integrates low rank representation and superpixel segmentation simultaneously, in which the HSI data is assumed to be lying in a low rank subspace within each homogeneous region of an estimated hidden field. First, the HSI data is projected into the Principal Component space, then the first principal component image is segmented into hundreds of homogeneous regions. Following, the spectral-only supervised Bayesian classifier, i.e., Sparse Multinomial Logistic Regression (SMLR), is utilized for estimating the likelihood probabilities of testing samples, then spatial information is exploited by low rank representation within each superpixel in a hidden field which is approximated to the pre-estimated likelihood probabilities. The proposed model can be easily solved by alternating direction method of multipliers (ADMM). Experimental results on real hyperspectral data, i.e., AVIRIS Indian Pines and ROSIS University of Pavia, show that the proposed classifier outperforms other state-of-the-art classifiers in terms of quantitative assessment and visual effect.

Due to the limitation of sensors and atmospheric conditions, hyperspectral images (HSI) are always contaminated by heavy noises, which significantly limits the subsequent applications. To mitigate the problem, this paper proposes a novel subspace spatial-spectral low rank learning method for hyper-spectral denoising. It is based on the assumption that spectra in HSI lie in a low-rank subspace and nonlocal spatial patches are self-similar. The spectral low-rank property is explored by decomposing the clean HSI into two sub-matrices of low rank and the spatial self similarity is exploited by weighed nuclear norm minimization in a nonlocal sense. The proposed restoration model is formulated into an iterative optimization model which can be effectively solved by a cyclic descent algorithm. Experimental results on both simulated and real HSI datasets show that the proposed method can significantly outperform the state-of-the-art methods in terms of quantitative assessment and visual quality.

This letter presents a novel mixed noise (i.e., Gaussian, impulse, stripe noises, or dead lines) reduction method for hyperspectral image (HSI) by utilizing low-rank representation (LRR) on spectral difference image. The proposed method is based on the assumption that all spectra in the spectral difference space of HSI lie in the same low-rank subspace. The LRR on the spectral difference space was exploited by nuclear norm of difference image along the spectral dimension. It showed great potential in removing structured sparse noise (e.g., stripes or dead lines located at the same place of each band) and heavy Gaussian noise. To simultaneously solve the proposed model and reduce computational load, alternating direction method of multipliers was utilized to achieve robust reconstruction. The experimental results on both simulated and real HSI data sets validated that the proposed method outperformed many state-of-the-art methods in terms of quantitative assessment and visual quality.

In compressive sensing (CS) of images or videos, a block-based sensing or recovery scheme can facilitate low-cost sampling or recovery in memory and computation. However, its recovery with small block size and small subrate suffers greatly from its lack of information of the measurement data essential to recover a unique solution among many candidates. This study, based on prior knowledge of the signal to be sensed, namely, the relative magnitude difference of signal entries, designs a weighting process to limit the solution space of the recovered signal and combines it with much simplified Landweber iterations to deliver a complete recovery algorithm, called iterative weighted recovery (IWR). We theoretically verify the performance of the proposed IWR, including error bound, convergence rate, and stopping criterion. Application of the proposed IWR to block-based CS of images or videos confirms the quality improvement of the recovered images or videos and reduction of recovery time.

In video coding, traditional motion estimation methods work well for videos with camera translational motion, but their efficiency drops for other motions, such as rotational and dolly motions. In this paper, a motion-information-based three-dimensional (3D) video coding method is proposed for texture plus depth 3D video. The synchronized global motion information of the camera is obtained to assist the encoder improve its rate-distortion performance by projecting the temporal neighboring texture and depth frames into the position of the current frame, using the depth and camera motion information. Then, the projected frames are added into the reference buffer list as virtual reference frames. As these virtual reference frames could be more similar to the current to-be-encoded frame than the conventional reference frames, the required bits to represent the residual will be reduced. The experimental results demonstrate that the proposed scheme enhances the coding performance for all camera motion types and for various scene settings and resolutions using H.264 and HEVC standards, respectively. With the computer graphic sequences, for H.264, the average gain of texture and depth coding are up to 2 dB and 1 dB, respectively. For HEVC and HD resolution sequences, the gain of texture coding reaches 0.4 dB. For realistic sequences, up to 0.5 dB gain (H.264) is achieved for the texture video, while up to 0.7 dB gain is achieved for the depth sequences.

This paper proposes a novel linear hyperspectral unmixing method based on l1-l2sparsity and total variation (TV) regularization. First, the enhanced sparsity based on l1-l2norm is explored to depict the intrinsic sparse characteristic of the fractional abundances in sparse regression unmixing model. By taking the correlation between hyperspectral pixels into account, total variation is minimized to enforce the spatial smoothness. Finally, the proposed model is solved by the extended alternating direction method of multipliers (ADMM). Experimental results on simulated and real hyperspectral datasets validate the excellent performances of the proposed method.

Owing to the existence of noise and intensity inhomogeneity in brain magnetic resonance (MR) images, the existing segmentation algorithms are hard to find satisfied results. In this study, the authors propose an improved fuzzy C ‐mean clustering method (FCM) to obtain more accurate results. First, the authors modify the traditional regularisation smoothing term by using the non‐local information to reduce the effect of the noise. Second, inspired by the mechanism of the Gaussian mixture model, the distance function of FCM is defined by using the form of certain exponential function consisting of not only the distance but also the covariance and the prior probability to improve the robustness. Meanwhile, the bias field is modelled by using orthogonal basis functions to reduce the effect of intensity inhomogeneity. Finally, they use the hierarchical strategy to construct a more flexibility function, which considers the improved distance function itself as a sub‐FCM, to make the method more robust and accurate. Compared with the state‐of‐the‐art methods, experiment results based on synthetic and real MR images demonstrate its accuracy and robustness.

Image denoising can reduce the perturbation inevitably generated during image signal acquisition and its subsequent processing. While the utilization of nonlocal properties can enhance the performance of the state-of-the-art denoising methods, a heavy computational burden is incurred especially for color images. Inspired by the high correlation in the texture information over color channels, for a reduction of the computational burden, this letter proposes denoising the luma channel first, and then, performing a patch-wise linear prediction to transfer the texture information of the denoised luma channel to the other two channels. The texture transferring is adapted to local characteristic (i.e., variance of the local patches) for a reduction of color smearing caused by large prediction error especially along edges. Experimental results confirm that the proposed method achieves performance improvement over the state-of-the-art color image denoising methods only at a slightly increased complexity of single-channel denoising.

While the rate-distortion optimized quantization (RDOQ) technique provides nontrivial coding gain in High Efficiency Video Coding (HEVC), it also involves considerable computations, whereby the complexity of quantization is significantly increased. In this paper, two schemes (the RDOQ bypass decision and the simplified level adjustment) are investigated to reduce the complexity of the quantization process in HEVC with RDOQ. The RDOQ bypass decision method initially selects the transform blocks for which the RDOQ is expected to give less/or no coding gain and enables the conventional uniform scalar quantization to be applied to these transform blocks instead of the RDOQ. The simplified level adjustment method only estimates the difference in rate-distortion costs among the candidate quantization levels to enable the encoder to select an optimal quantization level at a much reduced computational cost. Furthermore, the proposed simplified level adjustment scheme is designed so that it can be implemented in lookup tables. Experimental results show that the proposed fast method achieves 14.3% quantization complexity reduction in all intra main conditions, 15.2% in the random access main condition, and 14.9% in the low delay main condition on average with virtually no coding loss compared with the conventional quantization process with the RDOQ.

As a universal sampling procedure, compressive sensing (CS) considers that all samples of compressible signal are equally important. However, it is not true in in image/video signal since human visual system is more sensitive to low frequency components. Therefore, CS theory has been extended to hybrid and multi-scale CS to better capture the low-frequency samples. The computational complexity is another challenge in CS which can be solved by multi-resolution sensing matrix. In this paper, we propose a multi-scale/multi-resolution sensing matrix for Kronecker CS (KCS) based on separable wavelet transform and address measurement allocation problem with and without information of to-be-sensed image. The proposed methods not only perform better (3.72dB gain) but also low complexity and compatible with conventional reconstruction methods.

This paper proposes a compressive sensing of video (CSV) framework that utilizes statistical properties of video signal. The proposed scheme periodically acquires deterministic measurements (i.e., low frequency DCT coefficients) for key frames to improve recovering both key and nonkey frames. In addition, based on temporal correlation among frames, side information is generated for nonkey frames to model their important coefficients and sparsity in transform domain. This information helps better sensing by weighted sensing and measurement allocation. Experimental results show effectiveness of the proposed techniques and their significant improvement compared to prior work.

The existing non-local structure tensors utilize the isotropic nature of the neighborhoods and compare similarity of tensors by the Euclidean distance for tensor field regularization, thus resulting in limited performances in image analysis. In this paper, we present an anisotropic nonlocal tensor regularization method by using a directional projection based atomic decomposition scheme, which offers two advantages: better exploitation of spatial directional information for anisotropically regularizing tensor field, and straightforward employment of the Euclidean distance to compute smoothing weights without extending the original non-local means filter to tensor field. Experimental results show that the proposed anisotropic structure tensor is superior to existing representative nonlinear structure tensors, in terms of corner detection and image denoising.

In this paper, we propose a detail-preserving reconstruction method for total variation-based recovery in low subrate compressive sensing using cartoon texture image decomposition and residual reconstruction. It iteratively decomposes and reconstructs cartoon and texture image components separately. A nonlocal structure-preserving filter is utilized to reduce staircase artifacts while preserving nonlocal structures of image in the spatial domain. Experimental results show that the proposed method outperforms the conventional ones in terms of preserving small scale details of image.

Although total variation minimization technique is being widely used in compressive sensing recovery, it still suffers from the so called staircase artifact which is caused by losing fine details of image. As a solution for the problem, in this paper, we propose an edge-preserving weighting scheme utilizing nonlocal structure and histogram of natural image in the gradient domain. Experimental results show that the proposed scheme surpasses the traditional total variation and the edge-guided CS in both objective and subjective qualities.

Block-based compressive sensing is attractive for sensing natural images and video because it makes large-sized image/video tractable. However, its reconstruction performance is yet to be improved much. This paper proposes a new block-based compressive video sensing recovery scheme which can reconstruct video sequences with high quality. It generates initial key frames by incorporating the augmented Lagrangian total variation with a nonlocal means filter which is well known for being good at preserving edges and reducing noise. Additionally, local principal component analysis (PCA) transform is employed to enhance the detailed information. The non-key frames are initially predicted by their measurements and reconstructed key frames. Furthermore, regularization with PCA transform-aided side information iteratively seeks better reconstructed solution. Simulation results manifest effectiveness of the proposed scheme.

Compressive imaging can acquire image signal in an under-sampled (i.e., under Nyquist rate) representation called measurement. However, measurement compression still has an essential problem in its overall rate-distortion performance. In this paper, we propose a measurement prediction method in which the best predictor is directionally selected in order to reduce the entropy of measurement to be sent. Generally, the measurement prediction usually works well with a small block while the quality of recovery is known to be better with a large block. In order to overcome this dilemma, we propose to use a structural measurement matrix with which compressive sensing is done in a small block size but recovery is performed in a large block size. In this way, both prediction and recovery are expected to be improved at the same time. Experimental results show its superiority in measurement coding amounting up to bitrate reduction by 39 %.

Scalability in video coding is an effective functionality to serve various video contents at different levels of resolution and quality. Based on the High Efficiency Video Coding (HEVC) standard which achieves superior compression performance compared to the H.264/AVC, this paper proposes a new scalable coding method with a coding unit (CU) structure prediction technique for HEVC-based quality scalability. The CU structure in enhancement layer (EL) is differentially encoded using that of the basement layer (BL) as a predictor. The binary values describing the CU structure of BL are operated exclusive-OR (XOR) with those in EL at each CU depth and position. Compared to the simulcast coding, simulation without having any residual prediction technique verifies that the proposed method gains in bit-saving by 0.4% on average.

H.264/AVC applies an adaptive in-loop deblocking filter in order to remove blocking artifacts. The deblocking filter is adaptively controlled by using the boundary strength (BS) parameter. Although the filter achieves advantages in reducing blocking artifacts, there are two problems we should consider to improve the filtering performance. The first problem is that the filter applies just a strong and the strongest filter to filter intra coded blocks. The strong filter can remove blocking artifacts at smooth regions; however, it removes detail or sharpness of pictures at complex regions. The second problem is that H.264/AVC does not pay attention in removing corner outliers which appear when an edge of an object crosses a corner of a block. Therefore, corner outliers clearly appear and reduce the subjective quality of reconstructed pictures. In this paper, we propose an adaptive deblocking filter scheme which removes corner outliers, and which utilizes the intra prediction information of intra coded blocks to adaptively select the BS. Experimental results show that our proposed filter significantly enhances the subjective quality by removing blocking artifacts at smooth regions while maintaining details or sharpness at complex regions. In addition, the objective quality is improved, resulting in PSNR gains of up to 0.16 dB.

Recovery algorithm based on total variation (TV) has shown its capability to recover high quality image in compressive sensing by preserving edges well but not fine details and textures. Recently, to improve this deficiency, characteristics of natural images are further utilized by adding some regularization terms into its recovery problem. In these efforts, this paper proposes a new regularization exploiting nonlocal properties of image using the nonlocal means filter in the gradient domain instead of the spatial domain. The Split Bregman method is applied to solve a combination of total variation and a new regularization term under the framework of Kronecker compressive sensing. Numerical experiments with the proposed and related regularizations verify significant improvement of the proposed method in term of both objective and subjective qualities.

In this paper, we propose a fast decision method scheme to reduce encoder complexity of high efficiency video coding. It is an early detection of SKIP mode in one CU-level based on the differential motion vector (DMV) and coded block flag (CBF). Experimental results show that the encoding complexity can be reduced by up to 34.55% in random access (RA) configuration and 36.48% in low delay (LD) configuration with only a little bit of rate increment compared to the high efficiency video coding test model (HM) 4.0 reference software.

The distributed compressive video sensing (DCVS) poses itself as a very promising framework for future video coding on mobile devices due to its very low complexity at the encoder not only in sampling but also in compression. However its blocking artifacts and oscillatory artifacts especially in high-frequency components seriously degrade perceptual quality. In this paper, we try to solve both problems of the high-frequency oscillatory and blocking artifacts as a whole by a proposed deblocking filter. By differentiating those well-decoded blocks from poorly-decoded blocks using concept of reliability, the proposed method can improve DCVS framework in both objective and subjective qualities.

In this paper, a novel multiple description video coding scheme is proposed to insert and control the redundancy at macro block (MB) level. By analyzing the error propagation paths, the relative importance of each MB is determined. The paths, in practice, depend on both the video content and the adopted video coder. Considering the relative importance of the MB and the network status, an unequal protection for the video data can be realized to exploit the redundancy effectively. In addition, a simple and effective approach is introduced to tune the quantization parameter for the variable rate coding case. The whole scheme is implemented in H.264/AVC by employing its coding options, thus generating descriptions that are compatible with the baseline profile and extended profile of H.264/AVC. Due to its general property, the proposed approach can be employed for other hybrid video codecs. The results demonstrate the advantage of the proposed approach over other H.264/AVC multiple description schemes.

In this paper, we propose a new motion vector coding scheme using optimal predictor. To improve coding performance of motion vector data, the proposed encoder selects an optimal predictive motion vector that produces minimum bits for the given motion vector. The proposed decoder estimates the optimal predictive motion vector without additional information from encoder for indicating which predictor is used. Experimental results show that the proposed scheme is very effective in reducing coded motion vector data since it achieves BDBR (Bjøntegaard Delta Bit Rate) gain of about 4.46% on average compared to the KTA 2.0 software.

Intra coding has relatively low coding efficiency, however, it is an indispensable coding tool since it provides random access and low-complex encoding. A new intra prediction mode coding scheme is proposed in this paper to improve its coding efficiency. The proposed scheme can save bits used in representing intra prediction mode by selectively not transmitting intra prediction mode bits. If the proposed decoder can estimate the optimal intra prediction mode chosen by the encoder for itself, it performs intra coding and skips sending intra prediction mode; otherwise, it performs the same conventional intra coding as the H.264/AVC. Simulation results show that compared to the H.264/AVC, the proposed method improves coding efficiency for various video sequences.

Recent development of portable devices and prevalence of high-throughput communication infrastructures make the video encoding in portable devices on high demand. However, its computational complexity makes the implementation of real-time video encoder on portable devices extremely difficult. Many fast algorithms to solve the problems are not efficient from the view point of worst workload since they consider only reduction of average computational complexity. Moreover, since the amount of reduction of computational complexity highly depends on video sequence, fixed fast algorithms cannot always achieve their full potential for real-time encoding. In this paper, we analyze the complexity of H.264/AVC video coding tools, and develop two parameters for complexity control. Consequently, we design a power-aware complexity scalable encoding scheme implementable on embedded system. Using our target embedded system, the proposed method is verified to save power by about 50%.

The new video coding paradigm based on the Slepian-Wolf and Wyner-Ziv theorems, namely, Distributed Video Coding (DVC) is being actively studied as a light-weighted video encoding technique. The Wyner-Ziv coding scheme being the most representative DVC algorithm reconstructs an image by eliminating noises on side information using channel code. The quality of side information is an essential factor to achieve high compression performance of the Wyner-Ziv coding, however, conventional side information generation methods have problems when there are complex motions between frames. In this paper, a new side information generation method based on one-bit transform is proposed. The proposed encoder transmits edge information of original image in order to help motion estimation at the decoder. Thus, it is possible to perform more accurate decoder-side motion estimation.

Arising needs for extremely simple encoder motivate investigations on distributed video coding (DVC). The Wyner-Ziv coding, one of the representative DVC schemes, reconstructs video by correcting noise on side information using channel code. Therefore, its coding performance decisively depends on the amount of noise in the side information. However it is hard for a decoder to generate good side information, especially in case that frames are less correlated., Large amount of parity bits certainly helps the decoder to correct noise but it only comes with low compression ratio. In this paper, we propose a method to enhance the quality of the side information, with incrementally transmitted parity bits from encoder. Results of our experiments have verified average PSNR gain up to 0.8 dB.

This paper proposes a reversible visible watermarking algorithm to satisfy a new application scenario where the visible watermark serves as a tag or ownership identifier, but can be completely removed at the receiver end to resume the original image data. To achieve lossless recovery of the image, the proposed algorithm consists of two processing procedures: data hiding and watermark embedding. In the first procedure, we preserve the information of image region to be covered by the visible watermark in the uncovered image portion. In the second procedure, we embed the watermark based on a user-key-controlled embedding mechanism. The two distinct procedures are integrated into a secure watermarking system by a specially designed user key. With correct user keys, authorized users can completely remove the watermark and losslessly recover the original image.

The MPEG-4 Part-10 AVC/H.264 standard employs several powerful coding methods to obtain high compression efficiency. To reduce the temporal and spatial redundancy more effectively, motion compensation uses variable block sizes, and directional intra prediction investigates all available coding modes to decide the best one. When the rate-distortion (R-D) optimization technique is used, an encoder finds the coding mode having minimum R-D cost. Due to the large number of combinations of coding modes, the decision process requires extremely high computation. To reduce the complexity, we propose two methods: early SKIP mode decision and selective intra mode decision. In the simulation results, the proposed methods are verified to significantly reduce the entire encoding time by about 60% with only negligible coding loss

In this paper, we propose a reversible visible watermarking algorithm to satisfy a new application scenario where the visible watermark serves as a tag or ownership identifier, but can be completely removed to resume the original image data. It includes two procedures: data hiding and visible watermark embedding. In order to losslessly recover both the watermark-covered and nonwatermark-covered image contents at the receiver end, the payload consists of two reconstruction data packets, one for recovering the watermark-covered region, and the other for the nonwatermark-covered region. The data hiding technique reversibly hides the payload in the image region not covered by the visible watermark. To satisfy the requirements of large capacity and high image quality, our hiding technique is based on data compression and uses a payload-adaptive scheme. It further adopts error diffusion for improving subjective image quality and arithmetic compression using a character-based model for increasing computational efficiency. The visible watermark is securely embedded based on a user-key-controlled embedding mechanism. The data hiding and the visible watermark embedding procedures are integrated into a secure watermarking system by a specially designed user key

MPEG-4 Part-10 AVC/H.264 provides much improved coding efficiency. The powerful rate-distortion optimization for adaptive motion compensation employing variable block sizes investigates all possible coding modes of a block to decide the best coding mode. This achieves highest possible coding efficiency, but it only comes at much higher computational complexity. In this paper, we propose two new techniques-'early SKIP mode decision in P- and B-slices' and 'selective intra mode decision'-which can significantly reduce the computational complexity. Simulation results verify that the proposed method makes it possible to reduce the encoding time by about 66% without any perceivable loss in coding rate and quality.

LZW is one of the well-known lossless compression methods. Since it has several remarkable features such that coding is simple, prior analysis on source is unnecessary, and the whole code table is not sent to its decoder, LZW is widely used in many applications. GIF, TIFF and PDF (compressing images) are good examples of them. In spite of its reputation on compression, however, few data hiding methods are directly applied to LZW itself. This may be due to few redundancies remained in losslessly compressed data: therefore there is not enough available room for data hiding. The existing methods are lossy, however, lossless approach is preferred to the lossy one. In this paper, we propose the DH-LZW method that embeds data to source data in a lossless manner. Through this paper, modifiable elements of LZW for data hiding are introduced and a way to handle them is proposed. Experiments show very promising results.

The paper presents a new selective temporal error concealment algorithm (STEC) for error-corrupted H.264/AVC video bitstreams over the cdma2000 (or UMTS) air interface. The proposed algorithm performs selective temporal error concealment depending on whether a lost MB (macroblock) is in the background or foreground. It is shown that under the FMO coding mode of H.264/AVC, the proposed algorithm provides PSNR gain up to 1.18dB compared to the algorithm built into the H.264/AVC test model. In addition, the proposed STEC has an average PSNR improvement of 0.33dB compared with that under the N slice coding mode.

H.264 can be coded with 7 different block sizes for motion-compensation in the inter mode, and various spatial directional prediction modes in the intra mode. To achieve as high a coding efficiency as possible, the H.264 encoder employs a complex mode decision technique based on rate-distortion optimization. It calculates rate distortion cost (RDcost) of all possible modes to choose the best one having the minimum RDcost. Therefore, this mode selection method calls for high computational complexity. In order to reduce the complexity, we propose two techniques - 'early SKIP mode decision' and 'selective intra mode decision'. The simulation results show that without considerable performance degradation, the proposed methods reduce encoding time by 30% on average and save the number of computing rate-distortion costs by 72%.

The adaptive and powerful coding schemes in H.264 provide significant coding efficiency and some additional merits like error resilience and network friendliness. In spite of these outstanding features, it is not easy to implement H.264 codec as a real-time system due to its high requirement of memory bandwidth and intensive computation. Although the variable block size motion compensation using multiple reference frames is one of the key coding tools to bring about its main performance gain, it demands substantial computational complexity due to exhaustive search among all possible combinations of coding modes. Many existent fast motion estimation algorithms are not suitable for H.264 having variable motion block sizes. In this paper, we propose the motion field adaptive search using the hierarchical block structure based on the diamond search applicable to variable motion block sizes.

Matching pursuit was demonstrated to be useful especially in low-bit-rate video coding. However, the massive computation required for finding atoms hinders its use in practical applications. This paper provides a new method that can drastically reduce the computational load without any degradation in performance of matching pursuit. We compare vector norms based on the Schwarz inequality to preclude substantial number of dictionary functions without actually evaluating their inner products in atom search. Experimental results show that the number of inner product calculations is only about 20%-35% of the conventional separability-based fast methods.

In face-to-face video communication, we commonly use only a single camera placed on top of a monitor screen. This general configuration gives poor eye contact problem to decrease the feeling of natural conversation since the user stares at the monitor screen rather than directly staring at the camera lens. In this paper we present a new approach for natural feeling in video communication using image-based modeling and rendering techniques. Our facial modeling approach has two components. The first component is to estimate the eye position from an input image to find the gaze-correction angle and generate the basic model to represent the user's facial shape. The second component is a model-based shape approximation from motion which generates the user's simple facial shape to be used in rendering. To render a good eye-contact image, we propose 3D mesh warping technique, a method to rotate input image with the correction angle and the facial model. Our approach is effective and convenient since it uses the characteristic information about a facial scene. Preliminary experimental results with real facial image shows the enhanced naturalness which the face-to-face video communication has to offer.

This paper proposes an error detection technique using fragile watermarking. The fragile watermark is embedded in the least significant bits of the selected transform coefficients decided to balance between deterioration of PSNR value and error detection efficiency. The proposed method is usable without additional bits in the video bitstream and can be implemented very efficiently. This method will be useful in an error prone environment like a wireless channel.

Matching pursuit has been shown to be useful especially in low bit-rate video coding. However, one practical concern in its application is the massive computation required for finding dictionary elements. This paper provides a new method that can drastically reduce the computational load without any degradation of image quality. We use a simple distance comparison based on the Schwarz inequality to preclude a substantial number of dictionary elements without actually evaluating inner products. Experimental results show the number of inner product calculations about 20-30% of the separability-based method by Neff and Zakhor (see IEEE Trans. Circuits and Systems for Video Technology, vol.7, p.158-71, Feb. 1997).

This paper addresses a classification problem in which class definition through training samples or otherwise is provided a priori only for a particular class of interest. Considerable time and effort may be required to label samples necessary for defining all the classes existent in a given data set by collecting ground truth or by other means. Thus, this problem is very important in practice, because one is often interested in identifying samples belonging to only one or a small number of classes. The problem is considered as an unsupervised clustering problem with initially one known cluster. The definition and statistics of the other classes are automatically developed through a weighted unsupervised clustering procedure that keeps the known cluster from losing its identity as the "class of interest". Once all the classes are developed, a conventional supervised classifier such as the maximum likelihood classifier is used in the classification. Experimental results with both simulated and real data verify the effectiveness of the proposed method.

This paper proposes two decision fusion-based multitemporal classifiers, namely, the jointly likelihood and the weighted majority fusion classifiers, that are derived using two different definitions of the minimum expected cost. Without any overhead incurred by multitemporal processing, a user-selected conventional pixelwise classifier makes local class separately using each temporal data set, and the multitemporal classifiers make the global class decisions by optimally summarizing those local class decisions. The proposed weighted majority decision fusion classifier can handle not only the data set reliabilities but also the classwise reliabilities of each data set. Classification experiment using the jointly likelihood decision fusion with three remotely sensed Thematic Mapper (TM) data sets shows more than 10% overall classification accuracy improvement over the pixelwise maximum likelihood classifier.

This paper proposes a novel blocking artifacts reduction method based on the notion that the blocking artifacts are caused by heavy accuracy loss of transform coefficients in the quantization process. We define the block boundary discontinuity measure as the sum of the squared differences of pixel values along the block boundary. The proposed method compensates for selected transform coefficients so that the resultant image has a minimum block boundary discontinuity. The proposed method does not require a particular transform domain where the compensation should take place; therefore, an appropriate transform domain can be selected at the user's discretion. In the experiments, the scheme is applied to DCT-based compressed images to show its performance.

We propose a novel blocking artifacts reduction method in image coding based on the least square block discontinuity criterion. We define the "inner product on the block boundaries" for N/spl times/N 2-D functions to introduce the concept of the "boundary-orthogonal 2-D functions". The proposed post-processing approach attempts to find the coefficients of the boundary-orthogonal 2-D functions whose pel-by-pel sum is added to the blocky image to minimize the block discontinuity. Experimental results, using DCT-based compressed images, are provided to show the performance of the proposed scheme. In the experiments, as a special case of the proposed method, we use three terms of 2-D DCT functions which turn out to be simple but very effective in reducing blocking artifacts.

This paper proposes a simple and efficient realization of letter-box conversion for displaying 16:9 images on a 4:3 screen. Compared to the true 16:9 letter-box display where 4:3 vertical decimation is used, the new method yields 5:3 display aspect ratio (slightly wider than 16:9 in the vertical direction) by using 5:4 vertical decimation, and enables the decimation filter to be implemented exactly and easily without multipliers. An efficient hardware architecture is also shown for implementing the proposed 5:4 decimation; in case of progressive scan, if requires only two m-bit adders and two m-bit 2:1 multiplexers where m is the number of bits (typically 8) per pixel. Experiment shows little visual difference between the proposed letter-box conversion and the true 16:9 display with the 4:3 decimation.>

This correspondence proposes a fast Parzen density estimation algorithm that would be especially useful in nonparametric discriminant analysis problems. By preclustering the data and applying a simple branch and bound procedure to the clusters, significant numbers of data samples that would contribute little to the density estimate can be excluded without detriment to actual evaluation via the kernel functions. This technique is especially helpful in the multivariant case, and does not require a uniform sampling grid. The proposed algorithm may also be used in conjunction with the data reduction technique of Fukunaga and Hayes (1989) to further reduce the computational load. Experimental results are presented to verify the effectiveness of this algorithm.>

A contextual classifier which can utilize both spatial and temporal interpixel dependency contexts is investigated. After spatial and temporal neighbors are defined, a general form of maximum a posterior spatiotemporal contextual classifier is derived. This contextual classifier is simplified under several assumptions. Joint prior probabilities of the classes of each pixel and its spatial neighbors are modeled by the Gibbs random field. The classification is performed in a recursive manner to allow a computationally efficient contextual classification. Experimental results with bitemporal TM data show significant improvement of classification accuracy over noncontextual pixelwise classifiers. This spatiotemporal contextual classifier should find use in many applications of remote sensing, especially when the classification accuracy is important.>