Generative face video compression (GFVC) facilitates ultra-low bitrate video transmission by leveraging deep generative models. Among existing methods, Compact Feature Temporal Evolution (CFTE) compresses inter-frames excluding the first frame into a compact 4×4 representation, achieving a competitive trade-off between compression efficiency and reconstruction quality. However, CFTE relies heavily on features generated during encoding for temporal progression inference, which limits further bitrate reduction. To overcome this limitation, we propose a sparse compact feature representation (SCFR) framework for GFVC that exploits the intrinsic sparsity of facial depth maps. By preserving critical geometric structures while drastically reducing redundancy, the SCFR framework compresses inter-frames into a more compact 2×2 latent representation, thereby substantially lowering the required transmission bitrate. We further integrate a hybrid compressed sensing and neural reconstruction module to reconstruct facial depth maps from sparse measurements efficiently. The decoder employs a dual-branch architecture that combines optical flow estimation and multi-scale feature fusion, thus enhancing motion modeling and facial detail recovery. As shown in Tables 1 and 2, the proposed SCFR framework for GFVC achieves significant BD-rate savings of {77.74% (DISTS), 71.87% (LPIPS), 16.83% (PSNR), and 31.7% (SSIM)} over the VTM 22.2 anchor on the VoxCeleb and CFVQA datasets, confirming its superiority in both perceptual and distortion-based metrics.

Current machine learning research on image compression mainly focuses on developing neural network architectures and preparing training datasets with compressed input and raw uncompressed labels. This study originates from a fundamental question of whether using raw uncompressed data as labels is the best approach to compression models. To address this question, we employ less compressed data as labels, rather than raw uncompressed data, in training the compression model. Moreover, we investigate how learning incrementally between compressed datasets can produce higher quality images than using raw uncompressed images (ground truth) as labels. Based on the results, we propose Compression Artifact-Aware Incremental Machine Learning for JPEG image compression, called CAIML. For JPEG compression artifact removal, we adopt ARCNN [1], which has the simplest network architecture and allows for a clear assessment of performance changes caused by different label settings. In CAIML, the label dataset is constructed with JPEG quality settings in consistent incremental steps, enabling ARCNN to learn compression artifact reduction incrementally. Table 1 shows the results obtained using JPEG, ARCNN, CAIML with increments of 10 (CAIML-INC10), and CAIML with increments of 20 (CAIML-INC20) starting at a quality factor of$q=10$. Both CAIML variants outperform the original ARCNN. Among them, CAIML-INC10 achieves the highest PSNR, indicating the lowest distortion. The results indicate that when neural networks incrementally learn from less compressed data, they achieve higher quality than learning from the raw uncompressed data (ground truth). CAIML is expected to be applicable to various neural network-based compression models for both images and videos.11This work was conducted during Hanna Jung's internship at Xidian University.

In this paper, we propose a lightweight multiscale reference frame generation network for VVC inter coding, named LMRFG. Unlike the previous work [1], LMRFG does not employ the high-performance operation point (HOP) network as preprocessing for frame enhancement. LMRFG replaces the bidirectional motion estimation network with a dualbranch coordinated attention motion estimator (DBCA-ME), which integrates X-axis and Y-axis optical flows for accurate optical flow estimation. Moreover, LMRFG uses depthwise over-parameterized convolutional layer (DO-Conv) to reduce model complexity and minimize bitstreams while maintaining video quality. LMRFG adopts a quantization parameter (QP) distance-based training strategy that takes compressed data at higher QP as input and compressed data at lower QP as label for training, thus addressing the imbalanced QP gap between the compressed input and its uncompressed label. LMRFG is embedded between DPB and RPL to generate a new reference frame and replace the original reference frame in RPL. As shown in Table 1, LMRFG achieves average BD-rate gains of {RA: 4.31% (Y), 6.54% (U), 7.05% (V)} and$\{\text{LDB}: 3.81 \%(\mathrm{Y}), 8.90 \%(\mathrm{U}), 8.77 \%(\mathrm{V})\}$over the VTM_11.0-NNVC_10.0 (NN-tools ON) anchor, achieving state-of-the-art performance.

In this paper, we propose a Generative Face Video Compression (GFVC) extension of the Versatile Video Coding (VVC) standard, called VVC-GFVC. Unlike existing GFVC models that transmit face representations of key points and compact features for inter-frames via the Supplemental Enhancement Information (SEI) messages, the proposed VVC-GFVC framework employs a generic video coding standard for face video compression and does not require any additional SEI messages during transmission. As shown in Fig. 1, VVC-GFVC utilizes VVC to compress face videos at an ultra-low bitrate and employs intra-frame guided face generation on the decoder side to reconstruct the face videos. The intra-frame guided face generation is based on a generative model that uses the intra-frame compressed at a low QP (QP 22) as guidance and the interframes compressed at a high QP (QP 52) as the target for face generation. In the generative model, we propose a multi-step asymmetric Cycle Generative Adversarial Network (GAN) that performs high-fidelity reconstruction on severely degraded face videos while achieving remarkable compression efficiency. The multi-step asymmetric Cycle GAN ensures training stability and efficiency in face generation by reducing the randomness in the generation process. The encoder of VVC-GFVC is based on the VVC encoder, which encodes intra-frame at QP 22 and inter-frames at QP 52 under the LDB configuration. The decoder of VVC-GFVC combines the VVC decoder with intra-frame guided face generation to reconstruct the face video. Experimental results show that VVC-GFVC achieves BD-rate savings of$43.50 \%, 46.81 \%, 5.64 \%$and 4.24 % in DISTS, LPIPS, PSNR and SSIM, respectively, over the VVC anchor (VTM 22.2) on Class A and Class B test sequences.

The fixed-point arithmetic in video coding is essential to prevent numerical drift and guarantee codec reproducibility across different hardware platforms. In this paper, we propose a cross-platform unified reference frame synthesis network (URFS-Net) that supports both 32 -bit float-point and 16 -bit fixed-point representations under RA and LB configurations. Given two decoded frames from the decoded picture buffer (DPB), URFS-Net generates a new reference frame for insertion into the reference picture lists (RPL), thereby enhancing the VVC inter prediction. Specifically, URFS-Net comprises a hierarchical feature encoder for multi-scale context extraction, a scale-adaptive flow predictor for accurate motion estimation across scales, and a frame refiner to reduce artifacts caused by warping. In the fixed-point representation, both the network weights and the latent features of URFS-Net are represented in 16-bit integer precision through a static quantization scheme. We employ the SADL library, developed by JVET, to implement the fixed-point URFS-Net. The PyTorch-trained .pth model is first exported to an .onnx file, then converted to a float-point .sadl file, and finally transformed into a fixed-point .sadl file with appropriately quantized weights. Table 1 shows that the float-point and fixed-point URFS-Net models provide average Y-PSNR BD-rate gains of {RA: 3.56%; 2.51%} and {LB: 2.88%; 1.86%} over NNVC-14.0. Moreover, Table 2 shows that URFS-Net achieves superior BD-rate performance compared to the latest method JVET-AM0175 [1], while maintaining low complexity and runtime.

In this paper, we propose a neural network-based in-loop filter (NNLF) for versatile video coding (VVC) intra coding based on a dual-branch collaborative architecture. The proposed NNLF leverages spatial-frequency feature fusion to effectively learn multi-type input features and adopts a dual-branch collaborative architecture to process global quantization features from the quantization parameter (QP) map branch and local artifacts from the prediction/partition frame branch separately. In the former branch, the multi-scale refinement block, driven by wavelet transform convolution (WTConv), suppresses inter-block brightness differences by operating on frequency-decomposed subbands. In the latter branch, the progressive residual enhancement network restores texture discontinuities in the predicted regions through a dual-attention mechanism. Moreover, the adaptive fusion block alleviates input feature conflicts based on two pixel-wise attention maps by learning the global features from the QP map and the local features from the prediction and partition frames, enabling an effective fusion of complementary information. Experimental results show that the proposed NNLF achieves average Bjøntegaard Delta (BD)-rate gains of 8.43% (Y), 22.16% (U), and 23.00% (V) in peak signal-to-noise ratio (PSNR) over the VVC test model (VTM)-11.0_neural network-based video coding (NNVC)-3.0 anchor under All Intra (AI) configuration. The code is available at: https://github.com/Xidian-Media-Lab/Dual-Branch-NNLF .

It has been demonstrated that deep learning generates high-quality reference frames and achieves considerable gains in inter coding. Existing neural network-based reference frame generation (NN-RFG) methods are mainly built on convolutional neural networks (CNNs) with limited receptive fields and are applied to the Versatile Video Coding (VVC). In this paper, we propose Transformer-based reference frame synthesis for VVC inter coding, named TRFS. We introduce Transformer into NN-RFG to capture global contextual information and construct multi-scale feature pyramids for accurate optical flow estimation. TRFS operates between the decoded picture buffer (DPB) and reference picture lists (RPL) in VVC, generating new reference frames through spatiotemporal compensation between previously reconstructed frames. First, we present a hierarchical feature extractor based on parameter-efficient Transformer to capture global contextual information with different resolutions and construct multi-scale feature pyramids. Second, we design a weight-sharing optical flow estimator consisting of residual blocks and Transformers to progressively refine the bidirectional or unidirectional optical flows in a coarse-to-fine manner. Third, we employ a U-net frame enhancer equipped with a ConvNeXt variant to learn residuals from the input and warped frames while removing blurring distortion caused by backward warping. To train the TRFS network, we utilize a two-stage incremental learning strategy based on quantization parameter (QP)-distance to address the imbalanced QP gap between the compressed input and its label, enhancing its learning capability. Various experiments demonstrate that TRFS achieves average Bjøntegaard Delta rate (BD-rate) gains of {RA: 6.61%, 13.86%, 13.30%} and {LB: 6.10%, 13.92%, 13.14%} for {Y, U, V} components over VTM-11.0_NNVC-10.0 with Neural Network-Based Video Coding (NNVC)-tools disabled. Furthermore, when evaluated against VTM-11.0_NNVC-10.0 with NNVC-tools enabled, TRFS delivers average BD-rate gains of {RA: 4.43% and LB: 3.40%} for the luma component.

Generative face video compression (GFVC) aims to achieve ultralow bitrate face video communication via deep generative models. As the representative GFVC work, the Compact Feature Temporal Evolution (CFTE) model compresses the bitstream for inter frames to 4×4, and shows outstanding coding efficiency. However, the feature extraction suffers from difficulty in distinguishing foreground face and background, resulting in erroneous face generation. In this paper, we propose a GFVC model using depth estimation and compressed sensing. Depth maps provide prior information on the spatial face structure, while compressed sensing significantly reduces redundancy by leveraging low-dimensional sparse representation. Thus, we utilize depth maps to retain face geometry and reduce background interference in feature extraction, and combine them with compressed sensing to reduce feature redundancy and enhance face generation. The proposed GFVC encoder combines depth estimation and compressed sensing to perform compact and precise extraction of face features. Compacter bitstream requires less bandwidth during transmission, while more precise bitstream generates more accurate face images during decoding. The proposed GFVC decoder has a dual-branch structure based on optical flow estimation and multi-scale feature fusion to capture motion information and enhance the face reconstruction quality. Experimental results show that the proposed GFVC model achieves average BD-rate gains of 77.04% and 64.43% on official test sequences in JVET, i.e. Class A VoxCeleb and Class B CFVQA datasets, over VTM22.2 in terms of DISTS and LPIPS metrics, respectively. The code is available at https://github.com/Changhaocup/Depth_Estimation_and_Compressed_Sensing-GFVC.

In recent years, learned image compression (LIC) has made significant improvement in coding efficiency. However, existing LIC methods often ignore the extraction of local features, especially for repetitive textures. Moreover, they focus on feature extraction and compression in the spatial domain, overlooking the potential of frequency information. To address them, we propose a frequency aware LIC network that combines Swin Transformer and discrete wavelet transform (DWT). The encoder and decoder of the proposed LIC network has a symmetric network structure based on Swin Transformer to maintain the global feature extraction capability and guarantee the local feature extraction. To enhance the frequency learning ability, we introduce a frequency-aware module based on DWT in the hyper-prior network that captures context in the low-frequency domain and jointly learns spatial and frequency features. Since the training for LIC networks requires a lot of computing resources and time costs, we design a training strategy based on knowledge transfer to significantly improve the convergence speed and stability of training. The proposed training strategy optimizes parameter adjustment during training by reducing the number of iterations required for model training and avoiding overfitting, thereby significantly reducing training time and ensuring coding efficiency. Experimental results show that the proposed method achieves average BD-rate gains of 3.72% and 14.78% over VVC intra coding (VTM 11.0) on JPEG AI, Kodak24 and CLIC datasets in terms of PSNR and MS-SSIM, respectively.

The Joint Video Exploration Team (JVET) established an Ad Hoc Group on neural network-based video coding (NNVC), and launched the NNVC tools that integrated the low operating point (LOP) in-loop filter and NN-based intra prediction (NN-Intra) into the Versatile Video Coding (VVC). In this paper, we provide the combination test results of NNVC tools and NN-Inter in VVC to observe the complementarity and performance overlap between them. For the combination test, we use a Transformer-based reference frame synthesis (TRFS) network for VVC inter prediction, named NN-Inter. TRFS is integrated between the decoded picture buffer (DPB) and the reference picture lists (RPL) in VVC. Specifically, two adjacent decoded frames from the DPB are fed into TRFS to learn an intermediate frame, which is regarded as reference frame and inserted into the RPL for inter prediction enhancement. We set up four groups of experiments: "LOP + NN-Intra", "LOP + NN-Inter", "NN-Intra + NN-Inter", and "LOP + NN-Intra + NN-Inter". All experiments take VTM-11.0_NNVC-10.0 with NNVC tools disabled as the reference anchor. Experimental results show that the combination of LOP, NN-Intra, and NN-Inter achieves the best Bjøntegaard Delta rate (BD-rate) gains of {11.74% (Y), 19.57% (U), and 18.76% (V)} over the reference anchor. Although NN-Intra and NN-Inter demonstrate nearly additive gains due to their independent operations, significant performance overlap exists between LOP and NN-Inter due to the redundant enhancement functionality, especially for chroma channels.

Recent studies have shown that event data can effectively handle motion blur and preserve fine details in dynamic scenes, thereby significantly enhancing motion deblurring performance. In this paper, we propose event-guided motion deblurring with wavelet-based cross-modal feature fusion. We design cross-modal deblurring UNet to fuse event data and motion blurred image using discrete wavelet transform (DWT) and cross-modal feature fusion, named DUNet. DUNet performs multi-scale feature fusion based on a UNet architecture and dual-branch cross-modal Transformers to address both global and local blurs in a coarse-to-fine manner. Moreover, DUNet decomposes motion blurred image into four subbands based on DWT to align high-frequency subbands (LH, HL, HH) with the edge-focused event data for complementary fusion while leveraging the low-frequency subband (LL) to preserve global structure and semantic details. For effective fusion of event data and motion blurred image, the dual-branch cross-modal Transformers are employed to reduce computational complexity while effectively capturing global context. Various experiments on GoPro and REBlur datasets show that DUNet generates motion deblurring results with sharp edges, enhanced details, and reduced color distortion, outperforming existing state-of-the-art methods in terms of PSNR, SSIM, and computational complexity.

In low light conditions, visible (VIS) images are of a low dynamic range (low contrast) with severe noise and color, while near-infrared (NIR) images contain clear textures without noise and color. Multispectral fusion of VIS and NIR images produces color images of high quality, rich textures, and little noise by taking both advantages of VIS and NIR images. In this article, we propose the deep selective fusion of VIS and NIR images using unsupervised U-Net. Existing image fusion methods are afflicted with the low contrast in VIS images and flash-like effect in NIR images. Thus, we adopt unsupervised U-Net to achieve deep selective fusion of multiple scale features. Due to the absence of the ground truth, we use unsupervised learning by formulating an energy function as a loss function. To deal with insufficient training data, we perform data augmentation by rotating images and adjusting their intensity. We synthesize training data by degrading clean VIS images and masking clean NIR images using a circle. First, we utilize pretrained visual geometry group (VGG) to extract features from VIS images. Second, we build an encoding network to obtain edge information from NIR images. Finally, we combine all features and feed them into a decoding network for fusion. Experimental results demonstrate that the proposed fusion network produces visually pleasing results with fine details, little noise, and natural color and it is superior to state-of-the-art methods in terms of visual quality and quantitative measurements.

Since the quality of the reference frames is critical for VVC inter-coding, the neural network (NN)-based reference frame generation aims to generate a better quality reference frame from the two decoded frames in the decoded picture buffer (DPB) and inserts it into the reference picture list (RPL) for inter-prediction. However, it is difficult to directly apply video frame interpolation or extrapolation to the reference frame generation because compression artifacts severely degrade the quality of the generated reference frame. In this article, we propose a multiscale reference frame generation network for VVC inter-coding, named MRFGNet. To reduce the compression artifacts, MRFGNet adopts the high-performance operation point (HOP) network, which has been released by JVET, as preprocessing for frame enhancement. Unlike the HOP in-loop filter that takes multiple inputs, MRFGNet only takes the reconstructed frame and quantization parameter (QP) map as input for the HOP network. Moreover, a frame generation network is proposed to conduct accurate motion estimation based on directional optical flow. Thus, in both RA and LDB configurations, MRFGNet has the same architecture to achieve both bidirectional and unidirectional predictions. MRFGNet estimates optical flow at multiple scales with the same dimension, thereby leveraging scale-independent bidirectional optical flow prediction. An optical flow warping and fusion module is designed to two cascaded U-nets for flow feature maps and get two frames, and finally, the two frames are fused to generate a reference frame. Furthermore, a novel training strategy based on QP distance is utilized to optimize MRFGNet by taking compressed data with higher quality as label for training. Experimental results show that MRFGNet achieves average BD rate changes of {−5.10% (Y), −12.14% (U), −11.51% (V)} and {−5.98% (Y), −15.32% (U), −14.22% (V)} over VTM_11.0-NNVC_4.0 anchor under RA and LDB configurations, respectively, as well as outperforms the state-of-the art method for reference frame generation, i.e., JVET-AD0160, by {0.78% (Y), 1.42% (U), 1.44% (V)} and {2.82% (Y), 4.92% (U), 6.62 %(V)} under RA and LDB configurations, respectively.

Latent representation based on hyper-prior auto-encoders is recently applied to end-to-end image compression that shows comparable performance to the latest Versatile Video Coding (VVC) intra coding. The rate-distortion efficiency of image compression is greatly affected by the latent representation extracted by an auto-encoder. In this paper, we propose learned image compression using a long and short attention (LSA) module. We introduce the LSA module into an autoencoder to obtain accurate latent representation of the image. The LSA module improves ability of extracting global and local image features for the auto-encoder, thereby saving bit rate and achieving higher compression efficiency. We add two LSA modules in the encoding stage and the decoding stage to improve the encoding and decoding capabilities of the learned image compression network. Experiments on the JPEG AI dataset show that the LSA module successfully reconstructs image details and thus the proposed method achieves state-of-the-art performance in terms of multi-scale structural similarity (MS-SSIM). Moreover, the proposed method outperforms the state-of-the-art methods at a low bitrate in terms of peak signal-to-noise ratio (PSNR).

According to Rayleigh scattering, the density of fog is closely related to the distance of the target from the visible light camera. However, the near infrared (NIR) spectral band is robust to the interference problem but has a single channel without color. In this paper, we propose deep fusion of visible (VIS) and near infrared (NIR) images for registration and defogging using cross modal transformer. The proposed fusion network has two main features: 1) Cross modal transformer deals with the channel and spatial redundancy between two source images while retaining their complementary information; 2) Since the blue channel in VIS image can represent the amount of fog in the scene according to Rayleigh scattering, we normalize it to obtain a fog weight map. To synthesize training data, the distance between object and camera is required, thus we use depth estimation to obtain it. Then, we assign different amounts of fog to each image to consider various weather conditions. First, we extract multi-scale features from the source images. Second, we use the cross modal transformer to reflect cross-domain interaction between the source images in spatial and channel domains. Third, we generate the fusion weight for the source images according to the fog weight map and reconstruct a fused result based on it. Various experiments on the widely-used benchmark and real-world datasets show that the proposed method generates natural-looking defogging results with clear textures in distant regions while keeping the original tone in near regions. Moreover, the proposed method outperforms state-of-the-art fusion methods in term of both visual quality and quantitative measurements.

High resolution (HR) videos are easy to go beyond the band-width limit during transmission after encoding. Downsampling followed by upsampling is a well-known strategy for compressing HR videos with limited bandwidth. Versatile video coding (VVC) provides a reference picture resampling (RPR) functionality to consider the limited bandwidth. In this paper, we propose a convolutional neural network (CNN) filter for super-resolution (SR) with the RPR functionality in VVC. We design a lightweight SR network that combines CNN with the RPR functionality in VVC, called lightweight network of multi-level mixed scale and depth information with attention mechanism (LMSDANet). For LMSDANet, we provide a lightweight block of multi-mixed scale and depth information with attention (LMSDAB) to extract multi-scale and convolutional layer depth information while enhancing the representation of features. Compared with VTM-11.0_NNVC-2.0 anchor, LMSDANet achieves {-9.16% (Y), 17.03% (U), -7.61% (V)} and {-4.14% (Y), 6.34% (U), -2.25% (V)} BD-rate changes (average on A1 and A2) in AI and RA configurations, respectively.

In this paper, we propose a convolutional neural network (CNN) filter for reference picture resampling (RPR)-based super-resolution (SR) with wavelet decomposition. The proposed CNN filter takes the low resolution (LR) reconstructed frame (RecLR), LR prediction frame (PreLR) and high resolution (HR) RPR upsampled frame (RPRoutput) as the input for RPR-based SR. Thus, the proposed CNN filter not only learns a mapping function between LR and HR images, but also effectively removes blocking artifacts in the reconstructed frame. We adopt wavelet decomposition to make RPRoutputthe same size as RecLRand PreLRas well as obtain the relationship between high frequency (HF) and low frequency (LF) components. To maximize feature reuse under the limited parameters, we design a residual spatial and channel attention block (RSCB) that combines residual blocks with spatial attention and channel attention to learn the weighted local information and global information in different receptive fields. Experimental results show that the proposed CNN filter achieves -8.98% and -4.05% BD-rate reductions on Y channel in AI and RA configurations over VTM-11.0_NNVC-2.0 anchor, respectively.

Unlike RGB images, NIR images are robust to atmospheric environments such as Rayleigh scattering. In this paper, we propose long-range imaging using multispectral fusion of RGB and NIR images. The proposed network consists of feature extraction, feature fusion, and reconstruction. For feature extraction, we adopt pyramid feature selection to capture multi-scale information. For feature fusion, we present a learnable attention based fusion block (AFB) to refine and fuse latent features of RGB and NIR images and a multi-scale fusion block (MFB) to integrate the fused feature in multiple scales. For reconstruction, we use three convolutions corresponding to the encoder to reconstruct the fusion image. Besides, we perform image registration based on scale-invariant feature transform (SIFT) to align NIR image referring to RGB image. Experimental results show that the proposed network successfully recovers hidden textures lost in RGB images while keeping colors and outperforms state-of-the-art fusion methods in terms of both visual quality and quantitative measurements.

In versatile video coding (VVC), the in-loop filters suppress compression artifacts while reducing distortion. However, they have a limit of removing complicated compression artifacts due to the hand-crafted design. In this paper, we propose a convolutional neural network (CNN)-based in-loop filter for VVC intra coding. We introduce depthwise separable convolution and attention mechanism to make the proposed network lightweight and efficient. Moreover, we present two basic modules of residual attention block (RAB) and weakly connected attention block (WCAB) to extract and refine features. Besides, we design a multi-stage training strategy based on progressive learning to maximize the learning ability of the proposed network. Compared with VTM-11.0_NNVC-2.0 anchor, the proposed in-loop filter achieves average -7.08% (Y), -12.46% (U), -12.75% (V) BD-rate gains under All Intra (AI) configuration.

Depth completion densifies sparse depth images obtained from LiDAR and is a great challenge due to the given extremely sparse information. In this paper, we propose deep sparse depth completion using joint depth and normal estimation. There exists a mutually convertible geometric relationship between depth and surface normal in 3D coordinate space. Based on the geometric relationship, we build a novel adversarial model that consists of one generator and two discriminators. We adopt an encoder-decoder structure for the generator. The encoder extracts features from RGB image, sparse depth image and its binary mask that represent the inherent geometric relationship between depth and surface normal, while two decoders with the same structure generate dense depth and surface normal based on the geometric relationship. We utilize two discriminators to generate guide information for sparse depth completion from the input RGB image while imposing an auxiliary geometric constraint for depth refinement. Experimental results on KITTI dataset show that the proposed method generates dense depth images with accurate object boundaries and outperforms state-of-the-art ones in terms of visual quality and quantitative measurements.

In this paper, we propose a segmentation-guided fusion network for motion deblurring of long exposure images in low light condition, called SGFusionNet. SGFusionNet adopts multi-sensor fusion of of color (RGB) and near infrared (NIR) images to remove motion blur of long exposure images in low light condition. SGFusionNet first extracts foreground features from NIR images and background features from RGB images, and then fuses them at different scales through a fusion module. The fused features are fed into the decoding network for image reconstruction. For network training, we generate a dataset of RGB and NIR image pairs with a segmentation mask using JAI AD-130 GE camera. Due to the limit of generating ground truth, we design an unsupervised loss function based on the segmentation mask to constrain adaptive to foreground and background. Experimental results show that SGFusionNet effectively fuses fine NIR textures with RGB background with the guidance of the segmentation mask, while successfully removing motion blur in long exposure images.

Multi-sensor data has significant advantages over single sensor data due to the unique property of each sensor. We discover that multi-sensor collaboration can save bits and achieve stable compression performance according to quantization parameter (QP). In this paper, we propose a multi-sensor collaboration network for video compression based on wavelet decomposition, called MSCN. We introduce MSCN into 3D video coding based on color and depth sensors. The images acquired by a color sensor represent color and texture of the scene, while the images obtained by a depth sensor represent 3D geometric shape of the scene objects. Two sensor data are complementary, and color images help to reconstruct their corresponding depth images. First, we perform uniform sampling on the input depth video. Then, we compress the color and downsampled depth videos using 3D-HEVC codec. Finally, we reconstruct the depth video from the decoded color and depth videos by color guided depth super-resolution (SR). Experimental results show that MSCN achieves average BD-rate reductions of {−9.3%, −65.6%, −66.3%} and {−6.2%, −67.7%, and −69.2%} on 3D-HEVC test datasets for sampling factors 1, 2 and 4 in Random Access (RA) and All Intra (AI) configurations, respectively. Moreover, they verify that multi-sensor collaboration remarkably saves bits in video compression.

There exists spectral gap between visible (VIS) and near infrared (NIR) images causing dissimilar intensity according to reflection property of objects and materials. Therefore, it has a limit of applying traditional stereo matching to cross spectral disparity estimation. In this paper, we propose cross spectral disparity estimation from VIS and NIR paired images using disentangled representation and reversible neural networks. We build a supervised learning framework based on reversible blocks to extract scene features robust against the spectral gap. Reversible blocks decompose features into scene and style components to bridge the spectral gap between VIS and NIR images. We perform stereo matching on the scene component to get an initial disparity map by a 3D convolutional neural network. To generate clear edges in the disparity map, we use a semantic segmentation network as auxiliary information to refine the initial disparity map. Besides, to consider the lack of the ground truth, we synthesize reference disparity maps with guided image filtering. Experimental results demonstrate that the proposed method achieves accurate edges in disparity along object boundaries and outperforms the state-of-the-art methods in both visual comparison and quantitative measurements.

360-degree video provides omnidirectional views by a bounding sphere, thus also called omnidirectional video. For omnidirectional video, people can only see specific content in the viewport through head movement, i.e., only a small portion of the 360-degree content is exposed at a given time. Therefore, the viewport quality is of particular importance for 360-degree videos. In this article, we propose a quality enhancement of compressed 360-degree videos using viewport-based deep neural networks, named V-DNN. V-DNN is mainly composed of two modules: viewport prediction network (VPN) and viewport quality enhancement network (VQEN). VPN based on spherical convolution and 2D convolution generates potential viewports for omnidirectional video. VQEN takes the current viewport and its reference viewports as the input and enhances residual for the current viewport based on bidirectional offset prediction and Spatio-temporal deformable convolutions. Compared with HM16.16 baseline at QP = 37 under the Low Delay P (LDP) configuration, experimental results show that V-DNN achieves an average 0.605 dB and 0.0139 gains in viewport-based ΔPSNR and ΔMS-SSIM, respectively, and is 0.379 dB (59.63%) and 0.0073 (110.61%) higher than the multi-frame quality enhancement (MFQE-2.0) scheme at QP = 37, respectively. Moreover, V-DNN consistently outperforms MFQE-1.0, MFQE-2.0, and HM16.16 baseline at the other QPs in terms of ΔPSNR, ΔWS-PSNR, and ΔMS-SSIM.

In this paper, we propose selective progressive learning (SPL) for sparse depth completion. Given an intermediate feature map selected from the input RGB and sparse depth image pairs by the valid and invalid mask, SPL sequentially infers attention maps along two separate dimensions (spatial attention and channel attention), and then the attention maps are used to extract features for depth regression. First, we design sparsity selection module (SSM) to simplify the task and segment it as depth prediction from sparse depth guided by corresponding RGB pixels and RGB to depth translation. Second, we propose a guided attention module (GAM) to translate features from SSM to attention maps by spatial and channel transformations. We use GAM as guidance for two pathways in the depth prediction network. Third, we combine two pathways by three cascade regressions with quarter-size, half-size and full-size of the input. We use multi-scale learning to maximize the advantage of cascade regressions, which utilizes down-sampling to get the ground truth of their corresponding size. The SPL network structure based on sparsity greatly reduces the amount of calculation, and SPL successfully bridges the modality gap between RGB and depth images for sparse depth completion. Experimental results show that SPL generates competitive dense depth images with low complexity and outperforms state-of-the-art ones in terms of both accuracy and robustness. Ablation experiments demonstrate the generalization capability and stability of SPL.

In this paper, we propose low light image enhancement by multispectral fusion and convolutional neural networks (CNNs). We adopt multispectral fusion of color (RGB) and near infrared (NIR) images for low light image enhancement based on pyramid feature selection and attention map. The proposed fusion network consists of two subnetworks: denoising and fusion. Since low light RGB images contain severe noise with detail loss, we first utilize a denoising subnetwork to preprocess RGB images. In the denoising subnetwork, we use concat operation to prevent the loss of image features during training. We independently train the denoising subnetwork because denoising datasets are easy to obtain. After denoising, we use gamma correction to enhance the denoised low light image. Finally, we perform the fusion subnetwork for hidden texture recovery based on pyramid feature selection and attention map. To build the fusion subnetwork, we synthesize a low light image dataset based on smoothing and gamma correction. We generate the ground truth for training by adding the details of the NIR images into the smoothed RGB images. Experimental results show that the proposed fusion method generates high quality images with little noise, fine details and good colors.

Face generation from a rough sketch is a challenging task in computer vision, and most methods are based on conditional generative adversarial networks (cGANs) that use U-net as their backbone. They are helped by auxiliary information such as segmentation and parsing maps to generate visually pleasing facial images from sketches. However, it is hard to get the auxiliary information in reality when the input is only a rough sketch. In this paper, we propose deep face generation from a rough sketch using multi-level GANs. The proposed method does not need any auxiliary information for face generation except the input rough sketch. Instead, we generate a face image by the preceding generator and use it as auxiliary information for the succeeding generator along with the input sketch. The multilevel GANs progressively generates fine textures and contours in face images, thus leading to photo-realistic face components and hair even from a rough sketch. Various experiments show that the proposed method generates natural-looking face images and outperforms state-of-the-art methods in terms of visual quality and quantitative measurements.

We propose multispectral and multimodal image registration using a dynamic fusion index. The dynamic fusion index represents the fusion degree of two pair images, and we adopt it for multispectral and multimodal image registration. Moreover, adaptive weights represent confidence on structural features of pair images, and we utilize them to select reliable components of pair images for image registration. Based on the adaptive weights, we present adaptive weighted total variation to calculate the similarity between the reference and target images. In the adaptive weighted total variation framework, we first get the dynamic fusion index by fixing transformation parameters, and then estimate transformation parameters by fixing the dynamic fusion index. Finally, we determine the transformation parameters for image registration by iterating up to the convergence, and align the target image based on them. Experimental results demonstrate that the proposed method outperforms state-of-the-art registration ones in terms of visual quality and quantitative measurements.

Since a huge amount of datasets is required for semantic segmentation, few shot semantic segmentation has attracted more and more attention of researchers. It aims to achieve semantic segmentation for unknown categories from only a small number of annotated training samples. Existing models for few shot semantic segmentation directly generate segmentation results and concentrate on learning the relationship between pixels, thus ignoring the spatial structure of features and decreasing the model learning ability. In this paper, we propose a coarse-to-fine network for few shot semantic segmentation, named CFNet. Firstly, we design a region selection module based on prototype learning to select the approximate region corresponding to the unknown category of the query image. Secondly, we elaborately combine the attention mechanism with the convolution module to learn the spatial structure of features and optimize the selected region. For the attention mechanism, we combine channel attention with self-attention to enhance the model ability of exploring the spatial structure of features and the pixel-wise relationship between support and query images. Experimental results show that CFNet achieves 65.2% and 70.1% in mean-IoU (mIoU) on PASCAL-5ifor 1-shot and 5-shot settings, respectively, and outperforms state-of-the-art methods by 1.0%.

In this paper, we propose a convolutional neural network (CNN)-based post-processing filter for video compression with multi-scale feature representation. The discrete wavelet transform (DWT) decomposes an image into multi-frequency and multi-directional sub-bands, and can figure out artifacts caused by video compression with multi-scale feature representation. Thus, we combine DWT with CNN and construct two sub-networks: Step-like sub-band network (SLSB) and mixed enhancement network (ME). SLSB takes the wavelet subbands as input, and feeds them into the Res2Net group (R2NG) from high frequency to low frequency. R2NG consists of Res2Net modules and adopts spatial and channel attentions to adaptively enhance features. We combine the high frequency sub-band output with the low frequency sub-band in R2NG to capture multi-scale features. ME uses mixed convolution composed of dilated convolution and standard convolution as the basic block to expand the receptive field without blind spots in dilated convolution and further improve the reconstruction quality. Experimental results demonstrate that the proposed CNN filter achieves average 2.13%, 2.63%, 2.99%, 4.8%, 3.72% and 4.5% BD-rate reductions over VTM 11.0-NNVC anchor for Y channel on A1, A2, B, C, D and E classes of the common test conditions (CTC) in AI, RA and LDP configurations, respectively.

Recently, deep learning based object detectors have achieved good performance with abundant labeled data. However, data labeling is often expensive and time-consuming in real life. Therefore, it is required to introduce one shot learning into object detection. In this paper, we propose one shot object detection based on hierarchical adaptive alignment to address the limited information of one shot in feature representation. We present a multi-adaptive alignment framework based on faster R-CNN to extract effective features from query patch and target image using siamese convolutional feature extraction, then generate a fused feature map by aggregating query and target features. We use the fused feature map in object classification and localization. The proposed framework adaptively adjusts feature representation through hierarchical and aggregated alignment so that it can learn correlation between the target image and the query patch. Experimental results demonstrate that the proposed method significantly improves the unseen-class object detection from 24.3 AP50 to 26.2 AP50 on the MS-COCO dataset.

Emerging HDR videos enable the recording of adequate luminance information and representing realistic scenes to the audience. Due to the high precision of the HDR data recorded in the floating-point format, a quantization process is required to convert HDR data to integer data for compatibility with current transmission and display systems. In this study, a novel attention-aware quantization method is presented that attempts to preserve the contrast details in the region of interest of the human visual system. This method was applied in the context of HDR video coding. The proposed coding solution was compared with the current anchor solution in terms of the quality of the reconstructed video. Experimental results show that the proposed solution is able to improve the visual quality of encoded video with respect to the anchor solution. Additionally, the proposed solution achieves a bit-rate gain over the anchor with reference to the objective evaluation results.

Low light images suffer from a low dynamic range and severe noise due to low signal-to-noise ratio (SNR). In this paper, we propose joint contrast enhancement and noise reduction of low light images via just-noticeable-difference (JND) transform. We adopt the JND transform to achieve both contrast enhancement and noise reduction based on human visual perception. First, we generate a JND map based on an the human visual system (HVS) response model from foreground and background luminance, called JND transform. Second, for base image, we perform perceptual contrast enhancement based on luminance adaptation to effectively allocate a dynamic range to each gray level while preventing under enhancement (tone distortion) and over-enhancement. Third, we refine the JND map using Weber's law, luminance adaptation and visual masking. Weber's law enhances the JND map based on the luminance variation after contrast enhancement. Luminance adaptation suppresses noise for smooth regions, while visual masking enforces detail enhancement for textural regions. Fourth, we perform inverse JND transform to generate the enhanced luma channel from the JND map and base image. Finally, we conduct chroma denoising by transferring texture information of the enhanced luma channel to the chroma channels with guided filtering. Experimental results show that the proposed method achieves both contrast enhancement and noise reduction for low light images as well as outperforms state-of-the-art methods in terms of quantitative measurements.

Images captured in low light condition have a narrow dynamic range with a dark tone, which are seriously degraded by noise due to the low signal-to-noise ratio (SNR). Discrete wavelet transform (DWT) is invertible and thus is able to decompose an image into subbands without information loss minimizing redundancy. In this paper, we propose subband adaptive enhancement of low light images using wavelet-based convolutional neural networks. We adopt DWT to achieve joint contrast enhancement and noise reduction. We combine DWT with convolutional neural networks (CNNs), i.e. wavelet-based CNN, to facilitate subband adaptive processing. First, we decompose the input image into LL, LH, HL, and HH subbands to get low and high frequency components. Second, we perform contrast enhancement for LL subband and noise reduction for LH, HL and HH subbands. Finally, we perform refinement to enhance image details. Experimental results show that the proposed method enhances low light images while successfully removing noise as well as outperforms state-of-the-art methods in terms of visual quality and quantitative measurements.

The main challenge of face super-resolution is to overcome facial distortions in an upscaling process. Recent works have utilized facial priors such as facial landmarks and component maps to generate a precise super-resolved image. However, the facial priors are estimated from the ground-truth and deep neural networks. Thus, recent works based on the facial priors are not only limited to specific datasets including the ground-truth, but also need sub-networks to extract facial priors. To solve these problems, we propose a progressive face super-resolution network with non-parametric facial prior enhancement, called as NPFNet, which extracts and highlights facial components without any tricks, such as the ground-truth and deep neural networks. The self-enhancement module facilitates our network to fully utilize facial distinct features to enhance super-resolved images with a parameter-free operation. Extensive experiments on CelebA and VGGFace2 demonstrate that the proposed method outperforms state-of-the-art face super-resolution methods in terms of visual quality and quantitative measurements.

In this paper, we propose deep metric learning for human action recognition with SlowFast networks. We adopt SlowFast Networks to extract slow-changing spatial semantic information of a single target entity in the spatial domain with fast-changing motion information in the temporal domain. Since deep metric learning is able to learn the class difference between human actions, we utilize deep metric learning to learn a mapping from the original video to the compact features in the embedding space. The proposed network consists of three main parts: 1) two branches independently operating at low and high frame rates to extract spatial and temporal features; 2) feature fusion of the two branches; 3) joint training network of deep metric learning and classification loss. Experimental results on the KTH human action dataset demonstrate that the proposed method achieves faster runtime with less model size than C3D and R3D, while ensuring high accuracy.

Although most existing methods based on 3D mor-phable model (3DMM) need annotated parameters for training as ground truth, only a few datasets contain them. Moreover, it is difficult to acquire accurate 3D face models aligned with the input images due to the gap in dimensions. In this paper, we propose a face 2D to 3D reconstruction network based on head pose and 3D facial landmarks. We build a head pose guided face reconstruction network to regress an accurate 3D face model with the help of 3D facial landmarks. Different from 3DMM parameters, head pose and 3D facial landmarks are successfully estimated even in the wild images. Experiments on 300W-LP, AFLW2000-3D and CelebA HQ datasets show that the proposed method successfully reconstructs 3D face model from a single RGB image thanks to 3D facial landmarks as well as achieves state-of-the-art performance in terms of the normalized mean error (NME).

Face super-resolution (SR) has become an indispensable function in security solutions such as video surveillance and identification system, but the distortion in facial components is a great challenge in it. Most state-of-the-art methods have utilized facial priors with deep neural networks. These methods require extra labels, longer training time, and larger computation memory. In this paper, we propose a novel Edge and Identity Preserving Network for Face SR Network, named as EIPNet, to minimize the distortion by utilizing a lightweight edge block and identity information. We present an edge block to extract perceptual edge information, and concatenate it to the original feature maps in multiple scales. This structure progressively provides edge information in reconstruction to aggregate local and global structural information. Moreover, we define an identity loss function to preserve identification of SR images. The identity loss function compares feature distributions between SR images and their ground truth to recover identities in SR images. In addition, we provide a luminance-chrominance error (LCE) to separately infer brightness and color information in SR images. The LCE method not only reduces the dependency of color information by dividing brightness and color components but also enables our network to reflect differences between SR images and their ground truth in two color spaces of RGB and YUV. The proposed method facilitates the proposed SR network to elaborately restore facial components and generate high quality 8× scaled SR images with a lightweight network structure. Furthermore, our network is able to reconstruct an 128×128 SR image with 215 fps on a GTX 1080Ti GPU. Extensive experiments demonstrate that our network qualitatively and quantitatively outperforms state-of-the-art methods on two challenging datasets: CelebA and VGGFace2.

Reflection caused by glass often degrades the quality of an image and further makes it difficult to estimate depth. In this article, we propose joint reflection removal and depth estimation from a single image. We perform reflection removal (transmission recovery) and depth estimation jointly using a collaborative neural network that consists of four blocks: 1) encoder for feature extraction; 2) reflection removal subnetwork (RRN); 3) depth estimation subnetwork (DEN); and 4) depth refinement guided by the transmission layer. We achieve collaboration between reflection removal and depth estimation by concatenating intermediate features of DEN with RRN. Since the recovered transmission layer contains accurate edges of objects behind glass, we refine the estimated depth with its guidance by guided image filtering. The experimental results demonstrate that the proposed method achieves both reflection removal and depth estimation even for images with dominant reflections. Besides, this article offers a new way of treating reflections in images to introduce depth estimation into reflection removal and achieve reflection removal and depth estimation simultaneously.

Gradient-boosted decision trees (GBDTs) are widely used in machine learning, and the output of current GBDT implementations is a single variable. When there are multiple outputs, GBDT constructs multiple trees corresponding to the output variables. The correlations between variables are ignored by such a strategy causing redundancy of the learned tree structures. In this article, we propose a general method to learn GBDT for multiple outputs, called GBDT-MO. Each leaf of GBDT-MO constructs predictions of all variables or a subset of automatically selected variables. This is achieved by considering the summation of objective gains over all output variables. Moreover, we extend histogram approximation into the multiple-output case to speed up training. Various experiments on synthetic and real data sets verify that GBDT-MO achieves outstanding performance in terms of accuracy, training speed, and inference speed.

Low light images suffer from low dynamic range and severe noise due to low signal-to-noise ratio (SNR). In this paper, we propose joint enhancement and denoising of low light images via just-noticeable-difference (JND) transform. We achieve contrast enhancement and noise reduction simultaneously based on human visual perception. First, we perform contrast enhancement based on perceptual histogram to effectively allocate a dynamic range while preventing over-enhancement. Second, we generate JND map based on an HVS response model from foreground and background luminance, called JND transform. Then, we refine JND map using Weber's law and visual masking. Weber's law enhances the JND map based on the luminance variation after enhancement, while visual masking provides noise suppression for smooth regions and detail enhancement for texture regions. Finally, we conduct chroma denoising that transfers texture information of the denoised luma channel to the chroma channels by guided image filtering. Experimental results show that the proposed method achieves good performance in contrast enhancement and noise reduction while successfully preserving details.

In low light condition, color (RGB) images captured by camera contain much noise and loss of details and color. However, near infrared (NIR) images are robust to noise and have clear textures without color. In this paper, we propose multi-spectral fusion of RGB and NIR images using weighted least squares (WLS) and alternating guidance. Low light RGB images provide coarse image structure and color, while NIR images offer clear textures in a short distance. Since they are complementary, we adopt alternating guidance for fusion of RGB and NIR images based on WLS. First, we perform the first guidance for denoising on noisy RGB image to get base layer. Then, we conduct the second guidance for texture transfer on NIR image to get detail layer. Finally, we combine base and detail layers to produce a fusion result. We maximize the advantage of multispectral input (RGB and NIR) for fusion based on alternating guidance. Experimental results demonstrate that the proposed method achieves good performance in noise reduction, detail preservation and color reproduction as well as outperforms state-of-the-art ones in terms of quantitative measurement and computational efficiency.

In low light condition, the captured images suffer from severe noise and loss of textures and details due to low signal-to-noise ratio (SNR). In this paper, we propose low light image enhancement by multispectral fusion of color (RGB) and near infrared (NIR) images. We adopt adaptive weighted total variation to take the multispectral advantages in fusion. Since the NIR imaging depends on the NIR light strength, the NIR values are not reliable out of the range that NIR light can reach. Thus, based on a sigmoid function, we selectively perform total variation regularization according to the NIR values. If the NIR values are reliable, we perform the multispectral fusion with the guidance of the NIR image. If the NIR values are not reliable, i.e. none or too low, we only perform total variation regularization for denoising on the RGB image without NIR gradient transfer. For optimization, we convert the non-convex total variation regularization into a linear system using iterative reweighted least squares. Experimental results demonstrate that the proposed method successfully transfer the NIR details into the fusion result while removing noise and keeping colors as well as it produces natural looking fusion results even with unreliable NIR values.

In low light condition, the captured color (RGB) images are highly degraded by noise with severe texture loss. In this paper, we propose deep fusion of RGB and NIR paired images in low light condition using convolutional neural networks (CNNs). The proposed deep fusion network consists of three independent sub-networks: DenoisingNet, EnhancingNet, and FusionNet. We build a denoising sub-network to eliminate noise from noisy RGB images. After denoising, we perform an enhancing sub-network to increase the brightness of low light RGB images. Since NIR image contains fine details, we fuse it with the Y channel of RGB image through a fusion subnetwork. Experimental results demonstrate that the proposed method successfully fuses RGB and NIR images, and generates high quality fusion results containing textures and colors.

Near infrared (NIR) images are robust to ambient light and contain clear textures in low light condition. In this paper, we propose NIR image colorization using spatial adaptive denormalization (SPADE) generator and grayscale approximated self-reconstruction. Compared with traditional image to image translation methods, the proposed NIR colorization pursues photorealism rather than generative diversity. The challenge of this task is NIR-RGB mis-registration in training data. We address this problem by separately extracting NIR texture and RGB color with an end to end SPADE based model. Moreover, the proposed method facilitates a more precise synthesis with a given low light RGB reference image. Experiments on an open NIR-RGB dataset verify that the proposed method effectively preserves NIR textures and RGB colors in the synthesized results and outperforms the baselines in terms of visual quality and quantitative assessments.

Although near infrared (NIR) images contain no color, they have abundant and clear textures. In this paper, we propose deep NIR colorization with semantic segmentation and transfer learning. NIR images are capable of capturing invisible spectrum (700-1000 nm) that is quite different from visible spectrum images. We employ convolutional layers to build relationship between single NIR images and three-channel color images, instead of mapping to Lab or YCbCr color space. Moreover, we use semantic segmentation as global prior information to refine colorization of smooth regions for objects. We use color divergence loss to further optimize NIR colorization results with good structures and edges. Since the training dataset is not enough to capture rich color information, we adopt transfer learning to get color and semantic information. Experimental results verify that the proposed method produces a natural color image from single NIR image and outperforms state-of-the-art methods in terms of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

In this paper, we propose unsupervised person re-identification (ReID) using reliable and soft labels. We provide unsupervised person ReID to consider unknown pedestrian queries. We update ResNet model for person ReID based on reliable and soft labels. First, we perform unsupervised clustering on person images under different cameras, and select samples based similarity between images and cluster centers. Then, we conduct re-clustering for the selected samples and assign labels to them, i.e. reliable labels. We get probability of the unselected samples, i.e. soft labels. Finally, we update ResNet model for person ReID using reliable and soft labels. Experiments on Market-1501 and DukeMTMC-ReID demonstrate that the proposed method outperforms state-of-the-arts for unsupervised person ReID in terms of the cosine distance and accuracy.

In this paper, we propose super-pixel based finger earth mover's distance (SPFEMD) for hand gesture recognition. For finger representation, we design SPFEMD for similarity measurement between fingers and hand gestures, and use it as the distance metric for hand gesture recognition. First, we extract hands without any user interaction using both color and depth from Kinect camera. Then, we obtain the seed for hand segmentation on the depth map and segment hand on the color image using the seed. Since fingers contain distinct features for hand gesture recognition, we decompose the hand segment into palm and fingers based on morphological operation. Finally, we perform hand gesture recognition from fingers based on SPFEMD. Experiments on publicly available and our own data sets show the superiority of the proposed method over state-of-the-arts in terms of accuracy and confusion matrices.

Due to the structural inconsistency between color and depth, texture copying and edge blurring artifacts appear in color-guided depth super-resolution. In this paper, we propose alternately guided depth super-resolution using weighted least squares (WLS) and zero-order reverse filtering. We adopt WLS for alternating guidance, and alternately use color and depth as guidance in WLS to suppress texture copying artifacts. Since color guidance causes edge blurs in depth due to the mismatch between color and depth, we apply zero-order reverse filtering to depth images to alleviate edge blurring artifacts. Moreover, WLS is a global optimization-based filter and thus it is effective in removing depth noise. Experiments on Middlebury and real scene datasets show that the proposed method outperforms state-of-the-art methods in terms of quantitative and qualitative measurements.

In this paper, we propose multi-focus image fusion using light field data and convolutional neural networks (CNNs). We mainly investigate data augmentation for network training. We adopt light field data to generate all-clear images and focus maps for training by refocusing. We produce focus maps at the pixel level in fully convolutional networks from multi-focus images. This is different from other CNN-based fusion methods that treat multi-focus fusion as binary classification to produce focus maps at the patch level. To train the proposed networks, we construct a multi-focus image dataset which contains both multi-focus images and their ground truth (all-clear images and focus maps) using light field data. Experimental results show that the proposed method generates accurate focus maps close to the ground truth as well as outperforms state-of-the-art fusion methods in terms of quantitative measurements.

Facial image inpainting is a challenging task due to the loss of key components in faces such as eyes and nose. In this paper, we propose a simple yet efficient method to inpaint missing information from a facial image. We build an end-to-end multi-level generative network to capture features at various levels while reducing training time. We adopt multi-scale feature maps to produce natural-looking faces with realistic texture. To optimize the parameters of the proposed network, we use two kinds of losses: content and texture. The former loss consists of mean absolute error (MAE) and multi-scale structural similarity (MS-SSIM) losses to minimize distortion in content, while the latter one contains style and adversarial losses to facilitate texture synthesis. Experimental results demonstrate that the proposed method outperforms state-of-the-art methods in terms of both visual quality and quantitative measurements. The code is available at https://github.com/JieLiu95/MLGN.

Converting near-infrared (NIR) images into color images is a challenging task due to the different characteristics of visible and NIR images. Most methods of generating color images directly from a single NIR image are limited by the scene and object categories. In this paper, we propose a novel approach to recovering object colors from multi-spectral NIR images using gray information. The multi-spectral NIR images are obtained by a 2-CCD NIR/RGB camera with narrow NIR bandpass filters of different wavelengths. The proposed approach is based on multi-spectral NIR images to estimate a conversion matrix for NIR to RGB conversion. In addition to the multi-spectral NIR images, a corresponding gray image is used as a complementary channel to estimate the conversion matrix for NIR to RGB color conversion. The conversion matrix is obtained from the ColorChecker's 24 color blocks using polynomial regression and applied to real-world scene NIR images for color recovery. The proposed approach has been evaluated by a large number of real-world scene images, and the results show that the proposed approach is simple yet effective for recovering color of objects.

Block-based transform coding in its nature causes blocking artifacts, which severely degrades picture quality especially in a high compression rate. Although convolutional neural networks (CNNs) achieve good performance in image restoration tasks, existing methods mainly focus on deep or efficient network architecture. The gradient of compressed images has different characteristics from the original gradient that has dramatic changes in pixel values along block boundaries. Motivated by them, we propose gradient guided image deblocking based on CNNs in this paper. Guided by the gradient information of the input blocky image, the proposed network successfully preserves textural edges while reducing blocky edges, and thus restores the original clean image from compression degradation. Experimental results demonstrate that the gradient information in the input compressed image contributes to blocking artifact reduction as well as the proposed method achieves a significant performance improvement in terms of visual quality and objective measurements.

Color (RGB) images captured under low light condition contain much noise with loss of textures. Since near-infrared (NIR) images are robust to noise with clear textures even in low light condition, they can be used to enhance low light RGB images by image fusion. In this paper, we propose fusion of RGB and NIR images using robust spectral consistency (RSC) and dynamic gradient sparsity (DGS), called RSC-DGS. We build the RSC model based on a robust error function to remove noise and preserve color/spectral consistency. We construct the DGS model based on vectorial total variation minimization that uses the NIR image as the reference image. The DGS model transfers clear textures of the NIR image to the fusion result and successfully preserves cross-channel interdependency of the RGB image. We use alternating direction method of multipliers (ADMM) for efficiency to solve the proposed RSC-DGS fusion. Experimental results confirm that the proposed method effectively preserves color/spectral consistency and textures in fusion results while successfully removing noise with high computational efficiency.

Although color images are easily interfered by glass, depth images captured by infrared sensors are robust to reflection. In this letter, we propose multi-modal reflection removal using convolutional neural networks (CNNs). We build a multi-modal CNN for reflection removal to separate transmission from reflection using depth information. The proposed network consists of two sub-networks: image restoration and depth adaptation. Image restoration sub-network (IRN) recovers transmission layer from the input image with reflection, whereas depth adaptation sub-network (DAN) guides reflection removal of the IRN. Moreover, to extract image details for reflection removal, we present a multi-scale loss function that penalizes non-similarity for multi-scale outputs. Experimental results demonstrate that the proposed method is robust to dominant reflections and outperforms state-of-the-art methods in terms of both peak signal-to-noise ratio (PSNR) and structural similarity.

When people take a picture through glass, the scene behind the glass is often interfered by specular reflection. Due to relatively easy implementation, most studies have tried to recover the transmitted scene from multiple images rather than single image. However, the use of multiple images is not practical for common users in real situations due to the critical shooting conditions. In this paper, we propose single image reflection removal using convolutional neural networks. We provide a ghosting model that causes reflection effects in captured images. First, we synthesize multiple reflection images from the input single one based on ghosting model and relative intensity. Then, we construct an end-to-end network that consists of encoder and decoder. To optimize the network parameters, we use a joint training strategy to learn the layer separation knowledge from the synthesized reflection images. For the loss function, we utilize both internal and external losses in optimization. Finally, we apply the proposed network to single image reflection removal. Compared with the previous work, the proposed method does not need handcrafted features and specular filters for reflection removal. Experimental results show that the proposed method successfully removes reflection from both synthetic and real images as well as achieves the highest scores in PSNR, SSIM and FSIM.

Dilated convolutions support expanding receptive field without parameter exploration or resolution loss, which turn out to be suitable for pixel-level prediction problems. In this paper, we propose multiscale single image super-resolution (SR) based on dilated convolutions. We adopt dilated convolutions to expand the receptive field size without incurring additional computational complexity. We mix standard convolutions and dilated convolutions in each layer, called mixed convolutions, i.e., in the mixed convolutional layer, and the feature extracted by dilated convolutions and standard convolutions are concatenated. We theoretically analyze the receptive field and intensity of mixed convolutions to discover their role in SR. Mixed convolutions remove blind spots and capture the correlation between low-resolution (LR) and high-resolution (HR) image pairs successfully, thus achieving good generalization ability. We verify those properties of mixed convolutions by training 5-layer and 10-layer networks. We also train a 20-layer deep network to compare the performance of the proposed method with those of the state-of-the-art ones. Moreover, we jointly learn maps with different scales from a LR image to its HR one in a single network. Experimental results demonstrate that the proposed method outperforms the state-of-the-art ones in terms of PSNR and SSIM, especially for a large-scale factor.

Segmentation of cardiac MRI images plays a key role in clinical diagnosis. In the traditional diagnostic process, clinical experts manually segment left ventricle (LV), right ventricle (RV) and myocardium to obtain guideline for cardiopathy diagnosis. However, manual segmentation is time-consuming and labor-intensive. In this paper, we propose automatic segmentation and cardiopathy classification in cardiac MRI images based on deep neural networks. First, we perform object detection based on a YOLO-based network to get region of interest (ROI) from the whole sequence of diastolic and systolic MRI. Then, we obtain a pixel-wise segmentation mask automatically by feeding ROI into fully convolutional neural networks (FCN). Finally, we construct a fully connected network for cardiopathy diagnosis to decide a heart disease from the given MRI. Experimental results show that the proposed method successfully segments LV, RV and myocardium as well as achieves 90% accuracy in heart disease classification.

In this paper, we propose single depth image super-resolution using convolutional neural networks (CNN). We adopt CN-N to acquire a high-quality edge map from the input low-resolution (LR) depth image. We use the high-quality edge map as the weight of the regularization term in a total variation (TV) model for super-resolution. First, we interpolate the LR depth image using bicubic interpolation and extract its low-quality edge map. Then, we get the high-quality edge map from the low-quality one using CNN. Since the CNN output often contains broken edges and holes, we refine it using the low-quality edge map. Guided by the high-quality edge map, we upsample the input LR depth image in the TV model. The edge-based guidance in TV effectively removes noise in depth while minimizing jagged artifacts and preserving sharp edges. Various experiments on the Middle-bury stereo dataset and Laser Scan dataset demonstrate the superiority of the proposed method over state-of-the-arts in both qualitative and quantitative measurements.

In this paper, we propose deep feature embedding learning for person re-identification (re-id) using lifted structured loss. Although triplet loss has been commonly used in deep neural networks for person re-id, the triplet loss-based framework is not effective in fully using the batch information. Thus, it needs to choose hard negative samples manually that is very time-consuming. To address this problem, we adopt lifted structured loss for deep neural networks that makes the network learn better feature embedding by minimizing intra-class variation and maximizing inter-class variation. Extensive experiments on CUHK03, CUHKOI and VIPeR datasets demonstrate the superior performance of the proposed method over state-of-the-arts in terms of the cumulative match curve (CMC) metric.

It is required that input features are represented as vectors or scalars in machine learning for classification, e.g. support vector machine (SVM). However, real world data such as 2D images is naturally represented as matrices or tensors with higher dimensions. Thus, structural information of the data whose dimensions are more than two is not successfully considered. One typical structural information which is useful for the classification task is the spatial relationship of the nearby data points. In this paper, to leverage this kind of structural information, we propose a novel classification method which combines total variational (TV) regularization with SVM, called TV-SVM. Since TV achieves a local smoothing property by penalizing the local discontinuity of data, TV-SVM preserves better local structure than the original SVM due to TV regularization. We solve the objective function of TV-SVM via the alternating direction method of multipliers (ADMM) algorithm. Experimental results on image classification show that TV-SVM is competitive to the state-of-the-art learning method in both classification accuracy and computational complexity.

In this paper, we propose fully convolutional siamese fusion networks for object tracking. We adopt the fusion strategy of convolutional layers for object tracking to achieve good feature representation based on convolutional neural networks. Specifically, we fuse three convolutional layers of VGGNet based on normalized cross correlation (NCC). First, we use three convolutional layers of VGGNet as the basis for fusion, and reduce the size of the convolutional layers based on a convolution kernel. Then, we resize the convolutional layers to be the same size as the deconvolutional layers for layer fusion. Next, we fuse the three layers based on NCC between the target and search region, and produce the response map. Finally, we get the tracking result from the response map by the maximum response. Various experiments on large-scale data sets verify that the proposed method is robust to occlusion, deformation, motion blur, and background clutter as well as outperforms state-of-the-art trackers in terms of distance precision and overlap success.

In this paper, we propose patch-based stereo matching using 3D convolutional neural networks (CNN). We extract spatial color and disparity features simultaneously through 3D CNN. We treat stereo matching as multi-class classification that the classes are all possible disparity values. We first generate a large set of patches from stereo images for 3D CNN. Then, we get an initial disparity map through 3D CNN and refine it using color image guided filtering. The color image guided filtering minimizes outliers and refines edges in disparity without texture copying artifacts. Experimental results show that the proposed method successfully estimates disparity in smooth and discontinuity regions while preserving edges as well as outperforms state-of-the-arts in terms of average errors.

High Dynamic Range and Wide Color Gamut (HDR/WCG) video coding adopts chroma downsampling and upsampling for coding efficiency, which causes degradation in visual quality. In this paper, we propose weighted chroma downsampling and luma-referenced chroma upsampling for H-DR/WCG video coding. We adopt ICtCp color space instead of YCbCr color space due to its advantage in chroma data processing. We perform weighted chroma downsampling based on the difference between neighbouring pixels before encoding. Moreover, we produce weights for chroma upsampling based on the correlation between luma and chroma data after decoding. Experimental results show that the proposed method successfully preserves edges in HDR/WCG videos while minimizing color distortion as well as outperforms Anchor in terms of both Bjonteggard delta bitrate (BD-rate) and HDR video quality measure (HDR-VQM).

Although light field cameras record both spatial and angular information, their angular and spatial resolutions are limited when capturing light field data. Thus, it is required to synthesize virtual views. In this paper, we propose high-quality virtual view synthesis based on multi -loss convolutional neural networks (CNN). We adopt multi-loss function for view synthesis in both pixel and feature spaces to increase the angular resolution of light field data. We combine three losses of feature loss, edge loss, and mean squared error (MSE) loss into the multi -loss function. We learn the view synthesis function based on simple three layers of CNN. Experimental results show that the proposed method successfully produces virtual views from light field data and outperforms state-of-the-arts in terms of peak-signal-to-ratio (PSNR) and structural similarity (SSIM).

Since the signal-to-noise ratio (SNR) is low in a dark environment, the captured images are seriously corrupted by sensor noise. Moreover, the sensor noise is signal-dependent, and has different characteristics from Gaussian distribution. In this paper, we propose low light image denoising based on Poisson noise model and weighted total variation (TV) regularization. In the data fidelity term, we adopt Poisson noise model to consider characteristics of sensor noise. In the regularization term, we present TV regularization based on two weights: Ratio between intensity mean and variance, and edge directionality. The ratio between intensity mean and variance adjusts the smoothing degree, while the edge directionality preserves image details. Experimental results demonstrate that the proposed method effectively removes sensor noise in low light images as well as outperforms the-state-of-the-arts in terms of the naturalness image quality evaluator (NIQE).

Although depth cameras acquire depth in dynamic scenes, the captured depth images are often noisy and of low resolution. In this paper, we propose intensity-guided depth upsampling using edge sparsity and weighted L0 gradient minimization. We exploit the edge sparsity in the gradient domain to enhance the resolution and quality of depth images. First, we get mutual structure between intensity and depth using joint mutual-structure filtering. Second, we generate weights for L0 gradient minimization based on gradient and entropy of the intensity image, and upsample the depth image using weighted L0 gradient minimization. Finally, we refine the depth image using adaptive fast weighted median filtering. Experiments on Middlebury 2005 and real-world scene datasets confirm that the proposed method produces edge-preserving depth upsampling results and outperforms state-of-the-art methods in terms of accuracy.

In this paper, we propose multi-spectral fusion and denoising (MFD) of RGB and NIR images using multi-scale wavelet analysis. We formulate MFD of RGB and NIR images as a maximum a posterior (MAP) estimation problem in the wavelet domain. The direct fusion of noisy RGB and NIR image often leads to contrast attenuation due to the discrepancy between RGB and NIR images. Thus, we generate the wavelet scale map for fusion and denoising based on correlation between NIR and RGB wavelet coefficients. To consider local contrast and visibility of NIR data on RGB components, we provide the contrast preservation term for scale map estimation based on the local contrast and visibility. We use the regularization term to select high visibility and contrast of NIR wavelet coefficients in the scale map. Since noise generally appears in the high frequency band, we use gradients of NIR wavelet coefficients as the weight for weighted least square (WLS) smoothing in the scale map. Based on the wavlet scale map, we perform fusion and denoising of RGB and NIR wavelet coefficients. Experimental results show that the proposed method successfully performs fusion of RGB and NIR images with noise reduction and detail preservation as well as outperforms state-of-the-arts in terms of discrete entropy (DE) and feature-based blind image quality evaluator (FBIQE).

Image quality in mobile displays is considerably influenced by ambient light. In the daylight condition, images on mobile displays are darkly perceived by the human visual system due to the limited dynamic range of display, which causes loss of luminance and details. In this paper, we propose readability enhancement of displayed images under ambient light by enhancing both luminance and details. We design a weighted optimization framework, which contains the data term for luminance enhancement and the gradient term for detail enhancement. In the data term, we use an ambient light nonlinear intensity-transfer function considering display properties, ambient light, and image contents. In the gradient term, we employ a threshold versus intensity adaptation function based on the degree of ambient light adaptation to compensate for gradient distortions. We solve the optimization framework using a numerical solver and achieve image enhancement in displayed images. Experimental results demonstrate that the proposed method significantly improves readability of mobile displays under ambient light by enhancing luminance and details of images.

In this paper, we propose variational fusion of time-of-flight (TOF) and stereo data for depth estimation using edge-selective joint filtering (ESJF). ESJF is able to adaptively select edges for depth upsampling from the TOF depth map, stereo matching-based disparity map, and stereo images. We adopt ESJF to produce high-resolution (HR) depth maps with accurate edge information from low-resolution ones captured by the TOF camera. First, we measure confidences of TOF and stereo data based on a Gaussian function to be used as fusion weights. Then, we upsample the TOF depth map using ESJF and extract vertical and horizontal discontinuity maps from it. Finally, we perform variational fusion of TOF and stereo depth data guided by the discontinuity maps. Experimental results show that the proposed method successfully produces HR depth maps and outperforms the state of the art in preserving edges and removing noise.

In low light condition, the signal-to-noise ratio (SNR) is low and thus the captured images are seriously degraded by noise. Since low light images contain much noise in flat and dark regions, contrast enhancement without considering noise characteristics causes serious noise amplification. In this paper, we propose low light image enhancement based on two-step noise suppression. First, we perform noise aware contrast enhancement using noise level function (NLF). NLF is used to get a noise aware histogram which prevents noise amplification, and we use the noise aware histogram in contrast enhancement. However, the increase of intensity by contrast enhancement reduces the visibility threshold, which makes noise visible by human eyes. Second, we utilize a just noticeable difference (JND) model from luminance adaptation to suppress noise based on human visual perception. Experimental results show that the proposed method successfully enhances contrast in low light images while minimizing noise amplification.

In this paper, we propose surrounding adaptive tone mapping in displayed images under ambient light. Under strong ambient light, the displayed images on the screen are darkly perceived by human eyes, especially in dark regions. We deal with the ambient light problem in mobile devices by brightness enhancement and adaptive tone mapping. First, we perform brightness compensation in dark regions using Bartleson-Breneman equation which represents lightness effect on the image under different surrounding illuminations. Then, we perform adaptive tone mapping to reproduce the whole image under various ambient light conditions. Adaptive tone mapping combines human visual characteristics with a tone mapping operation considering ambient light influence. Experimental results demonstrate that the proposed method significantly enhances the readability of displayed images under different surrounding light conditions.

Since pedestrians in videos have a wide range of appearances such as body poses, occlusions, and complex backgrounds, pedestrian detection is a challengeable task. In this paper, we propose part-level fully convolutional networks (FCN) for pedestrian detection. We adopt deep learning to deal with the proposal shifting problem in pedestrian detection. First, we combine convolutional neural networks (CNN) and FCN to align bounding boxes for pedestrians. Then, we perform part-level pedestrian detection based on CNN to recall the lost body parts. Experimental results demonstrate that the proposed method achieves 6.83% performance improvement in log-average miss rate over CifarNet.

In this paper, we propose retinex-based perceptual contrast enhancement in images using luminance adaptation. We use the retinex theory to decompose an image into illumination and reflectance layers, and adopt luminance adaptation to handle the illumination layer which causes detail loss. First, we obtain the illumination layer using adaptive Gaussian filtering to remove halo artifacts. Then, we adaptively remove illumination of the illumination layer in the multi-scale retinex (MSR) process based on luminance adaptation to preserve details. Finally, we perform contrast enhancement on the MSR result. Experimental results demonstrate that the proposed method successfully enhances contrast in images while keeping textures in highlight regions.

In this paper, we propose complex form of local orientation plane (Comp-LOP) for object tracking. Comp-LOP is a simple but effective descriptor, which is robust to occlusion for object tracking. It effectively considers spatiotemporal relationship between the target and its surrounding regions in a correlation filter framework by the complex form, which successfully deals with the heavy occlusion problem. Moreover, scale estimation is performed to treat target scale variations for improving tracking accuracy. Besides, an appropriate size of the search region is determined without any manual adjustment by computing the entropy ratio of the target to background. Experimental results show that the proposed method achieves good performance in object tracking in terms of efficiency, accuracy, and robustness.

In this paper, we propose variational fusion of time-of-flight (TOF) and stereo data using edge selective joint filtering (ESJF). We utilize ESJF to up-sample low-resolution (LR) depth captured by TOF camera and produce high-resolution (HR) depth maps with accurate edge information. First, we measure confidence of two sensor with different reliability to fuse them. Then, we up-sample TOF depth map using ESJF to generate discontinuity maps and protect edges in depth. Finally, we perform variational fusion of TOF and stereo depth data based on total variation (TV) guided by discontinuity maps. Experimental results show that the proposed method successfully produces HR depth maps and outperforms the-state-of-the-art ones in preserving edges and removing noise.

In this paper, we propose naturalness-preserved tone mapping in images based on perceptual quantization (PQ). PQ is a transfer function based on Barten's contrast sensitive function (CSF) which represents human visual perception on luminance, and we adopt it to generate a limit curve for perceptual contrast enhancement. First, we obtain a limit curve in an image based on PQ transfer function to adjust the degree of contrast enhancement. Second, we redistribute the histogram using the limit curve and achieve perceptual contrast enhancement. Experimental results demonstrate that the proposed method effectively enhances contrast in images while successfully preserving naturalness.

Hand segmentation is a fundamental technology in computer vision, which is used as a pre-processing step for various applications such as human-computer interaction (HCI), sign language translation, and medical systems [1]. This demo provides interaction-free hand segmentation using Kinect camera. It is completely automatic, which can automatically segment foreground from the background and select the seed on the depth image for hand segmentation using Kinect camera. Compared with existing methods [2], main contributions are summarized as follows: (1) We perform automatic seed selection on the depth image by foreground segmentation and variable-scale circle detection. (2) We perform seed-based hand segmentation on the color image using Gibbs random field.

In this paper, we propose boundary-preserving depth upsampling without texture copying artifacts and holes. Since high-resolution (HR) color images contain many edges and textures, they inevitably cause artifacts such as texture copying when they are used in depth upsampling. Thus, we perform rolling guidance filtering on HR color images before depth upsampling to preserve object boundaries in depth maps while successfully removing texture copying artifacts. Moreover, upsampled depth maps are often corrupted by noise and contain holes along object boundaries. Thus, we utilize fast weighted median filtering after depth upsampling to remove noise and fill holes. Experimental results demonstrate that the proposed method achieves boundary-preserving depth upsampling results without texture copying artifacts and holes through visual and quantitative evaluations.

Perceptual quantization (PQ) for HDR video coding has good perceptual uniformity in the luminance range with the modest bit-depth. However, the dynamic range of HDR videos is not well-used in PQ according to contents, especially for chroma channels. Thus, there exist the wasted dynamic ranges in PQ which cause detail loss and color distortions. In this paper, we propose adaptive quantization-based HDR video coding with HEVC Main 10 Profile. We perform adaptive mapping for 16bit to 10bit conversion based on cumulative distribution function (CDF), i.e. adaptive quantization, instead of linear mapping. First, we obtain CDF of each video frame using histogram analysis. Then, we conduct adaptive quantization based on CDF according to video contents, and generate metadata for inverse adaptive conversion in decoder, i.e. 10bit to 16bit conversion. Experimental results demonstrate that the proposed method reconstructs high-quality HDR videos and achieves a significant improvement in performance in terms of visual assessment and quantitative measurement.

Although light field data provides abundant cues for depth estimation, light field depth estimation suffers from occlusion and uncertain edges. In this paper, we propose occlusion robust light field depth estimation using segmentation guided bilateral filtering. First, we calculate refocused images from light field data using digital refocusing. Second, we perform support vector machines (SVM) classification to classify occluded pixels and non-occluded pixels. Third, we conduct different cost estimations on occluded and non-occluded pixels, and remove noise by cost volume filtering. Finally, we perform segmentation-guided bilateral filtering to refine the depth map while preserving edges. Experimental results on both synthetic and our own data sets demonstrate that the proposed method achieves light field depth estimation robust to occlusion while successfully preserving edges.

In this paper, we propose readability enhancement of low light videos based on discrete wavelet transform (DWT). Captured videos under the low light condition have a narrow dynamic range (low contrast) with a dark tone as well as are highly corrupted by noise. We achieve both contrast enhancement and noise reduction in low light videos using wavelet coefficients. First, we perform normalization to stretch a dynamic range of an image. Then, we decompose the image into high-pass and low-pass sub-bands using DWT. Next, we perform de-noising and weak edge enhancement in the high-pass sub-bands (LH, HL, HH) and contrast enhancement in the low-pass sub-band (LL). Finally, we conduct color correction to compensate for color distortions caused by contrast enhancement. Experimental results demonstrate that the proposed method outperforms state-of-the-art ones in contrast enhancement, noise reduction, and color reproduction in terms of both subjective and objective evaluations.

SIFT flow adopts SIFT descriptor to find correspondence between two images. However, SIFT flow is not robust to scale and rotation for dense corresponding matching. In this paper, we propose moment-based dense correspondence matching which is robust to image variation. First, we apply Zernike moments to SIFT descriptor, i.e. Moments of Gradients (MoG). Then, we combine SIFT flow with MoG for dense correspondence matching. Experimental results show that the proposed method achieves better performance in correspondence matching than SIFT flow.

Existing contrast enhancement techniques, such as histogram equalization (HE) and optimal contrast-tone mapping (OCTM), construct a pixel mapping function based on probability. However, they often allocate a large dynamic range to smooth areas, thus preventing the allocation of dynamic range resources in some regions that people are more interested in. To deal with this problem, we propose optimized perceptual tone mapping (OPTM), which performs contrast enhancement of images considering human visual attention. First, we construct a saliency histogram based on human visual attention to quantitatively measure the degree of visual attention instead of a probability histogram. Then, we perform contrast enhancement of images subject to constraints, such as the maximum tone distortion allowed. Finally, we adjust a pixel mapping function based on a just-noticeable-distortion model to prevent overenhancement. The experimental results demonstrate that OPTM produces visually pleasing results without overenhancement in large smooth regions and achieves a significant performance improvement in contrast enhancement.

Image quality of mobile displays are significantly influenced by ambient light. In the daylight condition, displayed images on mobile displays are darkly perceived by human visual system (HVS), which suffer from significant detail loss. However, only luminance enhancement seriously affects image details especially for bright regions. To overcome this problem, we propose a quadratic optimization framework which includes data term for luminance enhancement and gradient term for detail enhancement. In the data term, we provide an ambient light nonlinear intensity-transfer function for adaptive luminance enhancement depending on display properties, ambient light, and image contents. In the gradient term, Weber's law is employed for detail enhancement. Finally, we achieve both luminance and detail enhancement by solving the optimization framework. Experimental results demonstrate that the proposed method remarkably enhances the visibility of displayed images under strong ambient light.

Since images captured under low light conditions have low dynamic range and are seriously degraded by noise, it is a challengeable task to achieve both contrast enhancement and noise reduction from low light images. In this paper, we propose a readability enhancement method of low light images based on dual-tree complex wavelet transform (DTCWT). We perform contrast enhancement and noise reduction for low light images based on wavelet coefficients. First, we conduct illumination compensation to contain fine details and fully utilize dynamic range. Then, we decompose the image into high-pass and low-pass sub-bands by DTCWT, and perform contrast limited adaptive histogram equalization (CLAHE) and a nonlinear transform in low-pass and high-pass sub-bands, respectively, to achieve both contrast enhancement and noise reduction. Finally, we perform color correction to deal with the color distortion problem caused by contrast enhancement. Experimental results demonstrate that the proposed method outperforms state-of-the-art ones in contrast enhancement, noise reduction, and color reproduction in terms of both subjective and objective evaluations.

Although self quotient image (SQI) has been popularly used for face recognition as a retinex-based based illumination normalization method, it suffers from strong shadows in faces with abrupt intensity change. In this paper, we propose SQI-based illumination normalization for face recognition based on discrete wavelet transform (DWT). To remove shadow edges while preserving features, we combine DWT with SQI into retinex-based illumination normalization. First, we divide a face image into four sub-bands using wavelet analysis to enhance vertical, horizontal, and diagonal edges. Then, we perform histogram truncation to remove very bright and dark regions. Finally, we conduct illumination normalization from SQI and the histogram truncation result based on the retinex theory. Experimental results demonstrate that the proposed method outperforms state-of-the-art retinex-based ones in terms of face recognition accuracy.

We propose perceptual contrast enhancement of dark images based on textural coefficients. The textural coefficient indicates textural degree of intensity and adaptively stretches the dynamic range in an image. First, we calculate gray level difference between a central pixel and its adjacent ones. Because some differences are obviously noticeable by human eyes, we only use unnoticeable differences to obtain the textural coefficient. We apply the just noticeable difference (JND) of the human visual system (HVS) to obtain the proper threshold. Then, we apply a Gaussian kernel to texture coefficients for avoiding excessive differences between adjacent ones. Finally, we perform optimal contrast tone mapping to obtain a mapping function. Experimental results show that the proposed method successfully enhances dark regions while avoiding over-enhancement in bright regions without halo artifact and tone distortion.

In this paper, we propose HEVC encoder optimization for HDR video coding based on perceptual block merging. We perform perceptual block merging for HEVC Main 10 Profile-based HDR video coding. First, we generate 10bit luminance adaptation (LA) for quantized HDR videos based on Barten contrast sensitivity function (CSF) model. Then, we obtain a free-energy based just-noticeable-difference (FEJND) map in each frame to represent the disorderly concealment effect. Next, we calculate average FEJND values of each coding unit (CU) and the frame from the FEJND map to get the perceptual Lagrange multiplier for rate-distortion optimization (RDO). Finally, we achieve perceptual block merging by minimizing RD costs based on the perceptual Lagrange multiplier. Experimental results demonstrate that the proposed method produces the block merging results more correlated with human visual perception, while successfully improving the visual quality of the reconstructed HDR videos.

In this paper, we propose a backward compatible opto-electronic transfer function (OETF) for high dynamic range (HDR) video coding based on rational quantization. Hybrid log gamma (HLG) OETF combines a gamma function and a logarithmic function to achieve display backward compatibility. We adopt a uniform rational mapping function (URMF) based on rational quantization to approximate the logarithmic function. UMRF is a fast and simple quantization technique, and thus we replace the logarithmic function in HLG with UMRF. Experimental results demonstrate that the proposed method produces more pleasing reconstruction results in the human visual system (HVS) and achieves better BD-rate performance and lower computational complexity than HLG.

In this paper, we propose tone-preserving contrast enhancement in images using rational tone mapping and constrained optimization. We first perform spike-removed histogram equalization (SRHE) for contrast enhancement to avoid over-enhancement caused by the conventional HE. We conduct rational tone mapping (RTM) with low-complexity to preserve the original tone continuity. Then, we formulate a constrained optimization problem consisting of the contrast enhancement and tone-preserving terms. By minimizing the objective function, we find the best trade-off between the two conflicting terms. Experimental results demonstrate that the proposed method enhances contrast in images while successfully preserving details, and outperforms state-of-the-art ones in both subjective and objective assessments.

Most existing contrast enhancement methods obtain the maximum clipping point by iterations under a given PSNR/SSIM value, and perform backlight scaling to maintain image quality. However, they need high computational complexity to determine the optimal clipping point and often cause over-saturation in images. In this paper, we propose backlight scaled contrast enhancement for LCDs without given PSNR/SSIM values using image key-based compression. We automatically determine the optimal clipping point for backlight scaling using image decomposition and image key-based compression. We first decompose an image into base and detail layers using bilateral filtering, and then determine the maximum clipping point using image key-based compression. Finally, we perform image compensation from the clipping point using gamma correction. Extensive experiments on various test images demonstrate that the proposed method minimizes over-saturation effects and enhances overall image quality in backlight scaling while significantly reducing power consumption.

Although high dynamic range (HDR) videos are a very fascinating media to represent real-world scenes, they need a huge amount of memory to store and transmit due to the high bit-depth. Thus, it is a major challenge for HDR video coding to efficiently compress them without sacrificing perceptual quality. Perceptual Quantizer (PQ) provides a solution to this problem, which is used as Anchor based on the HEVC Main 10 coding. However, PQ is not adaptive to HDR contents because of using a fixed mapping function from floating-point luminance to 10bit quantized luma. In this paper, we propose an adaptive transfer function based on PQ for HEVC Main 10 Profile-based HDR video coding, called adaptive PQ. Different from PQ, the proposed transfer function adaptively maps luminance to luma according to HDR contents. Thus, adaptive PQ is able to efficiently utilize the quantization interval. Moreover, adaptive PQ has better perceptual uniformity in the luminance range than PQ because of adaptively using the luminance range. Experimental results demonstrate that adaptive PQ achieves a significant performance improvement in HDR video coding over PQ in terms of visual quality and bit-rate.

Although a time-of-flight (ToF) depth camera provides depth information in dynamic scenes, captured depth images are often noisy and of low resolution. In this paper, we propose color-guided boundary-preserving depth upsampling based on L0gradient minimization to enhance both resolution and visual quality of depth images. Guided by the high-resolution color image, we perform optimization to preserve boundaries along objects. We apply the L0gradient to the regularization term, and compute its weights from both color and depth images. We optimize the objective function using alternative minimization and half-quadratic splitting. Experimental results on Middlebury 2005 and real-world scene datasets demonstrate that the proposed method produces boundary-preserving depth upsampling results and outperforms state-of-the-art ones in terms of accuracy.

The weighted least squares (WLS) filter is a well-known edge preserving smoothing technique, but its weights highly depend on the image gradients. Thus, it is hard to smooth local textures with large gradients by WLS filter. In this paper, we propose a structure tensor-based WLS (ST-WLS) filter to enhance the smoothing ability of WLS filter while successfully keeping object boundaries. We introduce structure tensor into WLS filter to extract local structures in an image and use them to guide the smoothing process. Specifically, we use structure tensor to measure isotropy of pixels. Then, we calculate the smoothing weights for pixels based on both their isotropy and gradients. Experimental results demonstrate that ST-WLS filter has strong ability of smoothing textures, especially for isotropic textures, while preserving boundaries and structures of objects.

Speed up robust features (SURF) image geometrical registration algorithm available tends to have a one‐to‐many association problem in feature association. One feature point in an image is associated with multiple feature points in another image, of which some or even all are mismatching points. Coordinates of these mismatching points are used to compute transformation matrix, making it difficult to obtain desirable registration. To solve this problem of SURF algorithm, super‐SURF image geometrical registration algorithm is proposed, in which information richness areas are selected to detect feature points and to implement feature points association. The degree of closeness of multiple feature points from the one‐to‐many feature point pairs is analysed in order to remove feature point pairs with larger errors and retain those with smaller errors. Transformation matrix is then determined with coordinates of feature point pairs retained. Registration image can be obtained after transformation of floating image. Experimental results indicate that super‐SURF image geometrical registration algorithm is of higher matching accuracy with less running time than SURF algorithm.

In stereo matching, it is a challengeable task to effectively deal with occlusions between objects, which often cause disparity errors along object boundaries. In this paper, we propose discontinuity-preserving stereo matching based on reliability. We first estimate an initial disparity map by segmentation-based adaptive weights. Then, we obtain reliability for each pixel in an image, which indicates the correctness of the estimated disparity based on the matching cost. Finally, we minimize disparity errors caused by occlusions between objects using reliability and produce discontinuity-preserving disparity maps. Experiments on the Middlebury data set show that the proposed method successfully preserves disparity discontinuities along object boundaries and outperforms the state-of-the-art stereo matching methods.

In this paper, we propose interactive image segmentation using adaptive constraint propagation (ACP), called ACP Cut. In interactive image segmentation, the interactive inputs provided by users play an important role in guiding image segmentation. However, these simple inputs often cause bias that leads to failure in preserving object boundaries. To effectively use this limited interactive information, we employ ACP for semisupervised kernel matrix learning which adaptively propagates the interactive information into the whole image, while successfully keeping the original data coherence. Moreover, ACP Cut adopts seed propagation to achieve discriminative structure learning and reduce the computational complexity. Experimental results demonstrate that the ACP Cut extracts foreground objects successfully from the background and outperforms the state-of-the-art methods for interactive image segmentation in terms of both effectiveness and efficiency.

In this paper, we propose semi-supervised bi-dictionary learning for image classification with smooth representation-based label propagation (SRLP). Natural images contain complex contents of multiple objects with complicated background, clutter, and occlusions, which prevents image features from belonging to a specific category. Therefore, we employ reconstruction-based classification to implement discriminative dictionary learning in a probabilistic manner. We jointly learn a discriminative dictionary called anchor in the feature space and its corresponding soft label called anchor label in the label space, where the combination of anchor and anchor label is referred to as bi-dictionary. The learnt bi-dictionary is utilized to bridge the semantic gap in image classification. First, SRLP constructs smoothed reconstruction problems for bi-dictionary learning. Then, SRLP produces the reconstruction coefficients in the feature space over the anchor to infer soft labels of samples in the label space. Experimental results demonstrate that the proposed method is capable of learning a pair of discriminative dictionaries for image classification in the feature and label spaces and outperforms the-state-of-the-art reconstruction-based classification ones.

In this paper, we propose superpixel matching-based depth propagation for 2D to 3D video conversion with joint bilateral filtering. First, we perform superpixel matching to estimate motion vectors of superpixels from the reference depth map instead of block matching. This is because a superpixel is a group of pixels which have similar characteristics and thus superpixel matching performs better than block matching in estimating motion vectors. Then, we conduct depth compensation based on the motion vectors to obtain the current depth map. However, the size of each superpixel is not exactly the same, which causes matching errors in the compensated depth map. Thus, we perform joint bilateral filtering to refine the depth map. Experimental results show that the proposed algorithm successfully performs depth propagation and produce high-quality depth maps for 2D-to-3D conversion.

We propose visual comfort assessment for stereoscopic 3D (S3D) images based on salient discomfort regions. Color-based saliency successfully represents visual attention because that the human visual system (HVS) focuses on the most salient region in an image. Disparity-based saliency effectively expresses visual comfort in S3D images. Based on two saliencies, we extract salient discomfort regions which mostly determine the overall visual comfort degree in S3D images. Then, we predict the visual comfort score from salient discomfort regions by a disparity feature vector which combines saliency-weighted disparity and maximum disparity. Finally, we provide visual comfort index maps of S3D images based on visual comfort scores, thus showing perceptually salient discomfort regions and their degrees. Experimental results demonstrate that the proposed method achieves significantly high accuracy in visual comfort assessment as compared to existing methods.

In this paper, we propose perceptual backlight scaling for low power liquid crystal displays (LCDs) based on visual saliency. Visual saliency represents human attention, which locates the objects of interest in the human visual system (HVS). We perform perceptual backlight scaling for LCDs from a given PSNR based on visual saliency. In the proposed method, visual saliency is obtained by the log-spectrum. First, we get the clipped point of an input image from the given PSNR, and determine its backlight dimming factor. Then, we decompose an image into base and detail layers using bilateral filtering, and perform pixel compensation for the base layer. Finally, we enhance the contrast of the compensated image using the saliency map and detail layer. Experimental results demonstrate that the proposed method successfully reproduces a global tone in an image while effectively preserving fine details.

When we see a scene, the visual saliency attracts our eyes at the first glance, and the human visual system (HVS) controls eye lens to focus on the salient object. Inspired by the HVS mechanism, we propose a fixation point-based image segmentation method using a random walk model, called Point-Cut. For a given image, we first adopt the visual saliency to find the regions that humans seldom or never fixate on, which are regard as background regions. Then, we segment the whole object regions where HVS focus the fixation using the superpixel based a random walk model. Experimental results show that the proposed method successfully segments foreground objects around the fixation point and achieves good performance even with the minimum user interaction comparable to state-of-the-art interactive image segmentation methods.

Semi-automatic 2D-to-3D conversion is preferred for its advantage of handling the trade-off between human participation and 3D conversion effects. In this paper, we propose a novel depth propagation algorithm for semi-automatic 2D-to-3D conversion based on an adaptive image-guided autoregressive (AR) model joint superpixel matching. Firstly, we perform superpixel matching to estimate motion vectors of superpixels. Then, we conduct depth compensation based on motion vectors to obtain the current depth map from the reference depth map. However, the size of each superpixel is not exactly the same, which causes matching errors in the compensated depth map. Thus, we employ an adaptive image-guided AR model to minimize matching errors, and obtain the final depth map by minimizing AR prediction errors. Experimental results demonstrate that the proposed method successfully performs depth propagation and produce high-quality depth maps for 2D-to-3D conversion.

To remove the information redundancy among blocks, quadtree-based block partitioning is used in High Efficiency Video Coding (HEVC). In this paper, we propose a perceptual block merging method for quadtree-based partitioning in HEVC based on the disorderly concealment effect in human perception. We segment a frame into orderly and disorderly regions using a free-energy based just-noticeable-difference (FEJND) model, and perceptually calculate the rate-distortion (RD) cost during the merging process from the visual sensitivity to distortions in different regions. Experimental results demonstrate that the proposed method achieves better rate-SSIM performance and make the merging results more correlated with human perception than HM12.0 anchor while maintaining its visual quality of the video sequences.

Tone mapping effectively produces high dynamic range images on low dynamic range displays. Global tone mapping is simple and fast because of using the same mapping curve for every pixel without considering its neighboring ones in the radiance map. However, it often loses fine details in an image with low dynamic range displays. In this paper, we propose detail-preserving tone mapping for low dynamic range displays with adaptive gamma correction. We first perform global logarithmic tone mapping to reproduce natural tone for low dynamic range displays. Although global logarithmic mapping effectively considers human perceptual abilities, it often loses some details especially in dark regions. Thus, we perform adaptive gamma correction to achieve local contrast enhancement. Experimental results demonstrate that the proposed method significantly enhances fine details of images as well as successfully preserves the overall image quality.

In this paper, we propose power-constrained backlight scaling for LCDs using brightness compensated contrast-tone mapping operation, called BCCTMO. First, we increase pixel values to compensate for the reduced brightness caused by backlight dimming for power saving through maintaining the perceived luminance. Then, we perform a contrast-tone mapping operation (CTMO) to reduce the information loss caused by the just mentioned brightness compensation and enhance contrast simultaneously. Finally, we conduct color scaling to reproduce natural colors in LCDs. Experimental results demonstrate that BCCTMO produces high-quality images and saving power compared with state-of-the-art methods.

In this paper, we propose content adaptive image superresolution (SR) using gradient consistency and anisotropic regularization. The gradient consistency term effectively suppresses visual artifacts such as ringing and preserves sharp edges in images while the anisotropic regularization term adaptively preserves the high frequency information according to the gradient magnitude. The complementary two terms are elaborately combined into the gradient-consistency-anisotropic-regularization (GCAR) prior for the SR reconstruction. The GCAR prior is very effective in preserving image details and recovering high frequency information. Moreover, the proposed SR method employs an effective feedback-control loop which contains content adaptive de-convolution, re-convolution, and pixel substitution, and the GCAR prior is utilized in the content adaptive de-convolution step. Extensive experiments on various test images demonstrate that the proposed method produces natural-looking SR results in terms of both visual quality and quantitative performance.

Depth-image-based-rendering (DIBR) has received much attention in recent years as a promising technology for 3DTV systems. However, holes are inevitable in DIBR during the view synthesis procedure because the scene area, which has been occluded in the reference image, become visible in the synthesized virtual view. In this paper, we propose dictionary based hole filling with the assistance of depth. Because holes generally come from background area, we first segment background from foreground with the assistance of depth. Then, we construct a dictionary for hole filling from background. Finally, we employ the dictionary to fill holes in the synthesized virtual view. Experimental results demonstrate that the proposed method achieves good performance in hole filling in terms of visual quality and quantitative measures.

Super-resolution (SR) reconstruction has been widely studied in recent years because of its promising applications in image and video processing. However, comparatively few studies on achieving high computational efficiency without losing performance have been made. In this paper, we provide parallelization of deconvolution-based image SR on a multi-core central processing unit (CPU) platform. We divide the SR problem in images into many sub-problems which are computationally independent of each other. The sub-problems are solved by mapping threads to different cores on a multi-core CPU platform. Experimental results demonstrate that the proposed method achieves superior performance in terms of both image quality and runtime.1

Semantic concept detection is a very promising way to manage huge amounts of personal contents. In this paper, we propose discriminative structure learning for semantic concept detection with graph embedding. We focus on the task of whole-image categorization and employ graphical model inference based semi-supervised learning (SSL) to detect the semantic category of an image. To effectively extract global features from images, we utilize the spatial pyramid image representation. Then, we perform data warping over the histogram intersection kernel-based graph to learn discriminative features and make image distributions more discriminative for both labeled and unlabeled images. By data warping, each cluster of images is mapped into a relatively compact cluster as well as clusters become well-separated. Moreover, we adopt low-rank representation (LRR) in the embedded space to capture the global discriminative structure from the learned features for label propagation due to its good ability of capturing the global structure of data distributions and robustness against noise and outliers. Finally, we design a smooth nonlinear detector on the captured global discriminative structure to effectively propagate the concepts of labeled images to unlabeled images. Extensive experiments are conducted on four publicly available databases to verify the superiority of the proposed method compared to the state-of-the-art methods.

With advances in 3DTV technology, video stereolization has attracted much attention in recent years. Although video stereolization can enrich stereoscopic 3D contents, it is hard to create good depth maps from monocular 2D videos. In this paper, we propose an automatic example-based video stereolization method with foreground segmentation and depth propagation, called EBVS. To consider both performance and computational complexity, we separately estimate depth maps according to the key and non-key frames. In the key frames, we first estimate an initial depth map based on examples from the RGB-D training data set, then refine it to preserve boundaries of foreground objects. In the non-key frames, we propagate the depth map of the key frame using motion compensation, and generate depth maps. Finally, we employ depth-image-based-rendering (DIBR) to generate stereoscopic views from 2D videos and their depth maps. Extensive experiments verify that the proposed EBVS produces visually pleasing and realistic stereoscopic 3D views from 2D videos.

The dictionary based super-resolution (SR) approach has received much attention in recent years because sparse representation is very effective for image restoration tasks. By sparse representation, original image patches are represented as a sparse linear combination of atoms in an over-complete dictionary. However, the dictionary based SR approach has some disadvantages that it produces some ringing artifacts especially along the object boundaries and is not effective in reconstructing images which contain the patterns with strong edge. In this paper, we improve the dictionary based SR using nonlocal total variation regularization. In the training stage, we jointly train two dictionaries, Dhand Dl, from the low-resolution (LR) and high-resolution (HR) training data sets by using KSVD algorithm as in conventional methods. In the synthesis stage, we obtain the sparse coefficient vector from the LR test image over the LR dictionary, and reconstruct SR image patches using the coefficient vectors. Then, we employ nonlocal total variation regularization to remove annoying ringing artifacts and recover the patterns with strong edge in images. Experimental results on various test images demonstrate that the proposed method is very effective in enhancing the dictionary based SR approaches in terms of quantitative performance and visual quality.

We provide a position-patch based face hallucination method using convex optimization. Recently, a novel position-patch based face hallucination method has been proposed to save computational time and achieve high-quality hallucinated results. This method has employed least square estimation to obtain the optimal weights for face hallucination. However, the least square estimation approach can provide biased solutions when the number of the training position-patches is much larger than the dimension of the patch. To overcome this problem, this letter proposes a new position-patch based face hallucination method which is based on convex optimization. Experimental results demonstrate that our method is very effective in producing high-quality hallucinated face images.

In this work, we demonstrate a 3D DMB player which provides users with realistic 3D services over terrestrial-digital multimedia broadcasting (T-DMB). Since the bit-budget for the transmission of additional data stream is strictly limited in the current state of T-DMB, the 3D services are based on depth-image-based rendering (DIBR) that the depth information as the additional data can be compressed efficiently. This demonstration presents a depth preprocessing procedure for reducing holes (disocclusion area) in transmitters and a fast rendering procedure using look-up-table (LUT) in receivers. By means of the 3D services, viewers can consumes realistic and exciting 3D contents in mobile 3D DMB players.

In this paper, we propose the text line refinement method based on the SVM (support vector machine) output score for accurate text localization. In general, SVM output scores for the verification of text candidates provide a measure of the closeness to the text. Up to the present, most researchers used the score for the verification of the text candidate region. However, we use the output score for refining the initial text localization results. By means of the proposed approach, we can obtain more accurate text localization results. The effectiveness and efficiency of the proposed method is validated by extensive experiments on a complex database containing 435 images.

In sports videos, text information provides valuable game information such as scores, players, and so on. This text information plays an important role for sports video understanding, summarization, and retrieval. These texts which can be used for video highlight or content search are called key captions. In this paper, we provide a robust detection method of key captions in sports videos. We also provide a dual binarization method easily segmenting texts with different color polarities (i.e. dark and bright texts) from the background in the key captions. From this text information, we create the efficient sports navigation system. By conducting experiments on a large database, the system performance is demonstrated to be accurate and robust.

In sports videos, text provides valuable information about the game such as scores and information about the players. This paper provides a golf navigation system based on this player information. In the case of baseball and soccer, the location of key captions such as scoreboard captions is generally fixed during the game. However, in golf, the location of the key captions containing player information is not fixed. These captions cannot be determined from fixed locations. Therefore, we use a color pattern of captions and its repetition property instead of the location property to determine these key captions. We also propose a dual binarization method easily segmenting texts with different color polarities (i.e. dark and bright texts) from the background in the key captions. From this text information, we provide each segment with player information and create the golf navigation system. By conducting experiments on a large database, the system performance is demonstrated to be accurate and robust.

In this paper, we propose stroke filter for text localization in video images. First, we give the definition of text, which is based on the analysis of previous methods and intrinsic characteristics of text. Secondly, the definition is realized in the form of stroke filter, which is elaborately designed based on local region analysis. We also discuss the relationship between the proposed stroke filter and the other related filters. Furthermore, stroke filter can be implemented in a fast way without convolution operation. The effectiveness and efficiency of stroke filter is validated by extensive experiments on a challenging database.

Most existing methods of text segmentation in video images are not robust because they do not consider the intrinsic characteristics of text. In this paper, we propose a novel method of text segmentation based on stroke filter (SF). First, we give the definition of text, which is realized in the form of stroke filter based on local region analysis. Based on stroke filter response, text polarity determination and local region growing modules are performed successively. The effectiveness of our method is validated by experiments on a challenging database.