The main function of depth completion is to compensate for an insufficient and unpredictable number of sparse depth measurements of hardware sensors. However, existing research on depth completion assumes that the sparsity - the number of points or LiDAR lines - is fixed for training and testing. Hence, the completion performance drops severely when the number of sparse depths changes significantly. To address this issue, we propose the sparsity-adaptive depth refinement (SDR) framework, which refines monocular depth estimates using sparse depth points. For SDR, we propose the masked spatial propagation network (MSPN) to perform SDR with a varying number of sparse depths effectively by gradually propagating sparse depth information throughout the entire depth map. Experimental results demonstrate that MPSN achieves state-of-the-art performance on both SDR and conventional depth completion scenarios. Codes are available at htt ps: / / github.com/jyjunmcl/MSPN_SDR

A novel algorithm, called semantic line combination detector (SLCD), to find an optimal combination of semantic lines is proposed in this paper. It processes all lines in each line combination at once to assess the overall harmony of the lines. First, we generate various line combinations from reliable lines. Second, we estimate the score of each line combination and determine the best one. Experimental results demonstrate that the proposed SLCD outperforms existing semantic line detectors on various datasets. More-over, it is shown that SLCD can be applied effectively to three vision tasks of vanishing point detection, symmetry axis detection, and composition-based image retrieval. Our codes are available at https://github.com/Jinwon-Ko/SLCD.

In recent interactive segmentation algorithms, previous probability maps are used as network input to help predictions in the current segmentation round. However, despite the utilization of previous masks, useful information contained in the probability maps is not well propagated to the current predictions. In this paper, to overcome this limitation, we propose a novel and effective algorithm for click-based interactive image segmentation, called MFP, which attempts to make full use of probability maps. We first modulate previous probability maps to enhance their represen-tations of user-specified objects. Then, we feed the modulated probability maps as additional input to the segmentation network. We implement the proposed MFP algorithm based on the ResNet-34, HRNet-18, and ViT-B backbones and assess the performance extensively on various datasets. It is demonstrated that MFP meaningfully outperforms the existing algorithms using identical backbones. The source codes are available at hups.//github.com/cwleetul/Ml-P.

A novel approach to blind image quality assessment, called quality comparison network (QCN), is proposed in this paper, which sorts the feature vectors of input images according to their quality scores in an embedding space. QCN employs comparison transformers (CTs) and score pivots, which act as the centroids of feature vectors of similar-quality images. Each CT updates the score pivots and the feature vectors of input images based on their ordered correlation. To this end, we adopt four loss functions. Then, we estimate the quality score of a test image by searching the nearest score pivot to its feature vector in the embedding space. Extensive experiments show that the proposed QCN algorithm yields excellent image quality assessment performances on various datasets. Furthermore, QCN achieves great performances in cross-dataset evaluation, demonstrating its superb generalization capability. The source codes are available at https://github.com/nhshin-mcl8/QCN.

A novel clustering algorithm for orderable data, called unsupervised order learning (UOL), is proposed in this paper. First, we develop the ordered $k$-means to group objects into ordered clusters by reducing the deviation of an object from consecutive clusters. Then, we train a network to construct an embedding space, in which objects are sorted compactly along a chain of line segments, determined by the cluster centroids. We alternate the clustering and the network training until convergence. Moreover, we perform unsupervised rank estimation via a simple nearest neighbor search in the embedding space. Extensive experiments on various orderable datasets demonstrate that UOL provides reliable ordered clustering results and decent rank estimation performances with no supervision. The source codes are available at https://github.com/seon92/UOL.

Trit-plane coding enables deep progressive image compression, but it cannot use autoregressive context models. In this paper, we propose the context-based trit-plane coding (CTC) algorithm to achieve progressive compression more compactly. First, we develop the context-based rate reduction module to estimate trit probabilities of latent elements accurately and thus encode the trit-planes compactly. Second, we develop the context-based distortion reduction module to refine partial latent tensors from the trit-planes and improve the reconstructed image quality. Third, we propose a retraining scheme for the decoder to attain better rate-distortion tradeoffs. Extensive experiments show that CTC outperforms the baseline trit-plane codec significantly, e.g. by -14.84% in BD-rate on the Kodak loss less dataset, while increasing the time complexity only marginally. The source codes are available at https://github.com/seungminjeon-github/CTC.

A novel 4K video frame interpolator based on bilateral transformer (BiFormer) is proposed in this paper, which performs three steps: global motion estimation, local motion refinement, and frame synthesis. First, in global motion estimation, we predict symmetric bilateral motion fields at a coarse scale. To this end, we propose BiFormer, the first transformer-based bilateral motion estimator. Second, we refine the global motion fields efficiently using blockwise bilateral cost volumes (BBCVs). Third, we warp the input frames using the refined motion fields and blend them to synthesize an intermediate frame. Extensive experiments demonstrate that the proposed BiFormer algorithm achieves excellent interpolation performance on 4K datasets. The source codes are available at https://github.com/JunHeum/BiFormer.

A novel algorithm to detect road lanes in videos, called recursive video lane detector (RVLD), is proposed in this paper, which propagates the state of a current frame recursively to the next frame. RVLD consists of an intra-frame lane detector (ILD) and a predictive lane detector (PLD). First, we design ILD to localize lanes in a still frame. Second, we develop PLD to exploit the information of the previous frame for lane detection in a current frame. To this end, we estimate a motion field and warp the previous output to the current frame. Using the warped information, we refine the feature map of the current frame to detect lanes more reliably. Experimental results show that RVLD outperforms existing detectors on video lane datasets. Our codes are available at https://github.com/dongkwonjin/RVLD.

A novel continuous-mask-aware transformer for image inpainting, called CMT, is proposed in this paper, which uses a continuous mask to represent the amounts of errors in tokens. First, we initialize a mask and use it during the self-attention. To facilitate the masked self-attention, we also introduce the notion of overlapping tokens. Second, we update the mask by modeling the error propagation during the masked self-attention. Through several masked self-attention and mask update (MSAU) layers, we predict initial inpainting results. Finally, we refine the initial results to reconstruct a more faithful image. Experimental results on multiple datasets show that the proposed CMT algorithm outperforms existing algorithms significantly. The source codes are available at https://github.com/keunsoo-ko/CMT.

A seam is a set of pixels with minimum energy forming a continuous line in an image. By eliminating or duplicating seams iteratively, an input image can be retargeted. However, this process often results in blurring, stretching, or distortion problems around the seams, especially when extending a target image. We propose a novel approach for image extension using content-aware seam restoration to solve this problem. First, we design CSR-Net, which employs features from the horizontal region of target pixels to restore the seams. Second, we develop an image extension scenario based on the seam restoration and the training methodology of CSR-Net. Experimental results demonstrate that the proposed algorithm provides more accurate expanded results at seam pixels the seams than conventional algorithms.

A novel algorithm to detect road lanes in the eigen-lane space is proposed in this paper. First, we introduce the notion of eigenlanes, which are data-driven descriptors for structurally diverse lanes, including curved, as well as straight, lanes. To obtain eigenlanes, we perform the best rank-M approximation of a lane matrix containing all lanes in a training set. Second, we generate a set of lane candi-dates by clustering the training lanes in the eigenlane space. Third, using the lane candidates, we determine an optimal set of lanes by developing an anchor-based detection net-work, called SIIC-Net. Experimental results demonstrate that the proposed algorithm provides excellent detection performance for structurally diverse lanes. Our codes are available at https://github.com/dongkwonjin/Eigenlanes.

We propose the deep progressive image compression using trit-planes (DPICT) algorithm, which is the first learning-based codec supporting fine granular scalability (FGS). First, we transform an image into a latent tensor using an analysis network. Then, we represent the latent tensor in ternary digits (trits) and encode it into a compressed bitstream trit-plane by trit-plane in the decreasing order of significance. Moreover, within each trit-plane, we sort the trits according to their rate-distortion priorities and transmit more important information first. Since the compression network is less optimized for the cases of using fewer tritplanes, we develop a postprocessing network for refining reconstructed images at low rates. Experimental results show that DPICT outperforms conventional progressive codecs significantly, while enabling FGS transmission. Codes are available at https://github.com/jaehanlee-mcl/DPICT.

Novel contour descriptors, called eigencontours, based on low-rank approximation are proposed in this paper. First, we construct a contour matrix containing all object boundaries in a training set. Second, we decompose the contour matrix into eigencontours via the best rank-M approximation. Third, we represent an object boundary by a linear combination of the$M$eigencontours. We also incorporate the eigencontours into an instance segmentation framework. Experimental results demonstrate that the proposed eigencontours can represent object boundaries more effectively and more efficiently than existing descriptors in a low-dimensional space. Furthermore, the proposed algorithm yields meaningful performances on instance segmentation datasets.

A novel ordinal regression algorithm, called moving window regression (MWR), is proposed in this paper. First, we propose the notion of relative rank ($\rho$-rank), which is a new order representation scheme for input and reference instances. Second, we develop global and local relative regressors ($\rho$-regressors) to predict$\rho$-ranks within entire and specific rank ranges, respectively. Third, we refine an initial rank estimate iteratively by selecting two reference instances to form a search window and then estimating the$\rho$-rank within the window. Extensive experiments results show that the proposed algorithm achieves the state-of-the-art performances on various benchmark datasets for facial age estimation and historical color image classification. The codes are available at https://github.com/nhshin-mcl/MWR.

A novel future frame extrapolation algorithm using the future cost volume is proposed in this work. First, we develop the future cost volume to estimate motion vectors from a future frame to input frames. Second, we generate two future frame candidates by back-ward warping the input frames using the future motion vectors. Finally, we develop a synthesis network, which aggregates the two candidates to reconstruct the future frame faithfully. Experimental results demonstrate that the proposed algorithm significantly outperforms state-of-the-art extrapolators on various datasets.

A novel approach to rank estimation, called geometric order learning (GOL), is proposed in this paper. First, we construct an embedding space, in which the direction and distance between objects represent order and metric relations between their ranks, by enforcing two geometric constraints: the order constraint compels objects to be sorted according to their ranks, while the metric constraint makes the distance between objects reflect their rank difference. Then, we perform the simple $k$ nearest neighbor ($k$-NN) search in the embedding space to estimate the rank of a test object. Moreover, to assess the quality of embedding spaces for rank estimation, we propose a metric called discriminative ratio for ranking (DRR). Extensive experiments on facial age estimation, historical color image (HCI) classification, and aesthetic score regression demonstrate that GOL constructs effective embedding spaces and thus yields excellent rank estimation performances. The source codes are available at https://github.com/seon92/GOL

A lightweight blind image denoiser, called blind compact denoising network (BCDNet), is proposed in this paper to achieve excellent trade-offs between performance and network complexity. With only 330K parameters, the proposed BCDNet is composed of the compact denoising network (CDNet) and the guidance network (GNet). From a noisy image, GNet extracts a guidance feature, which encodes the severity of the noise. Then, using the guidance feature, CDNet filters the image adaptively according to the severity to remove the noise effectively. Moreover, by reducing the number of parameters without compromising the performance, CDNet achieves denoising not only effectively but also efficiently. Experimental results show that the proposed BCDNet yields state-of-the-art or competitive denoising performances on various datasets while requiring significantly fewer parameters.

We propose a novel guided interactive segmentation (GIS) algorithm for video objects to improve the segmentation accuracy and reduce the interaction time. First, we design the reliability-based attention module to analyze the reliability of multiple annotated frames. Second, we develop the intersection-aware propagation module to propagate segmentation results to neighboring frames. Third, we introduce the GIS mechanism for a user to select unsatisfactory frames quickly with less effort. Experimental results demonstrate that the proposed algorithm provides more accurate segmentation results at a faster speed than conventional algorithms. Codes are available at https://github.com/yuk6heo/GIS-RAmap.

A novel algorithm to detect an optimal set of semantic lines is proposed in this work. We develop two networks: selection network (S-Net) and harmonization network (H-Net). First, S-Net computes the probabilities and offsets of line candidates. Second, we filter out irrelevant lines through a selection-and-removal process. Third, we construct a complete graph, whose edge weights are computed by H-Net. Finally, we determine a maximal weight clique representing an optimal set of semantic lines. Moreover, to assess the overall harmony of detected lines, we propose a novel metric, called HIoU. Experimental results demonstrate that the proposed algorithm can detect harmonious semantic lines effectively and efficiently. Our codes are available at https://github.com/dongkwonjin/Semantic-Line-MWCS.

Recently, the encoder-decoder and intensity transformation approaches lead to impressive progress in image enhancement. However, the encoder-decoder often loses details in input images during down-sampling and up-sampling processes. Also, the intensity transformation has a limited capacity to cover color transformation between low-quality and high-quality images. In this paper, we propose a novel approach, called representative color transform (RCT), to tackle these issues in existing methods. RCT determines different representative colors specialized in input images and estimates transformed colors for the representative colors. It then determines enhanced colors using these transformed colors based on the similarity between input and representative colors. Extensive experiments demonstrate that the proposed algorithm outperforms recent state-of-the-art algorithms on various image enhancement problems.

We propose a novel loss weighting algorithm, called loss scale balancing (LSB), for multi-task learning (MTL) of pixelwise vision tasks. An MTL model is trained to estimate multiple pixelwise predictions using an overall loss, which is a linear combination of individual task losses. The proposed algorithm dynamically adjusts the linear weights to learn all tasks effectively. Instead of controlling the trend of each loss value directly, we balance the loss scale — the product of the loss value and its weight — periodically. In addition, by evaluating the difficulty of each task based on the previous loss record, the proposed algorithm focuses more on difficult tasks during training. Experimental results show that the proposed algorithm outperforms conventional weighting algorithms for MTL of various pixelwise tasks. Codes are available at https://github.com/jaehanlee-mcl/LSB-MTL.

We propose a novel video frame interpolation algorithm based on asymmetric bilateral motion estimation (ABME), which synthesizes an intermediate frame between two input frames. First, we predict symmetric bilateral motion fields to interpolate an anchor frame. Second, we estimate asymmetric bilateral motions fields from the anchor frame to the input frames. Third, we use the asymmetric fields to warp the input frames backward and reconstruct the intermediate frame. Last, to refine the intermediate frame, we develop a new synthesis network that generates a set of dynamic filters and a residual frame using local and global information. Experimental results show that the proposed algorithm achieves excellent performance on various datasets. The source codes and pretrained models are available at https://github.com/JunHeum/ABME.

A novel light field super-resolution algorithm to improve the spatial and angular resolutions of light field images is proposed in this work. We develop spatial and angular super-resolution (SR) networks, which can faithfully interpolate images in the spatial and angular domains regardless of the angular coordinates. For each input image, we feed adjacent images into the SR networks to extract multi-view features using a trainable disparity estimator. We concatenate the multi-view features and remix them through the proposed adaptive feature remixing (AFR) module, which performs channel-wise pooling. Finally, the remixed feature is used to augment the spatial or angular resolution. Experimental results demonstrate that the proposed algorithm outperforms the state-of-the-art algorithms on various light field datasets. The source codes and pre-trained models are available at https://github.com/keunsoo-ko/ LFSR-AFR.

We propose an optimized color contrast enhancement algorithm for dichromats with color vision deficiency using local and global information. Based on the fact that dichromats perceive colors projected onto a 2D plane, we first measure the color differences on the plane. Then, we formulate an optimization problem to minimize the perceived color differences subject to constraints on the projected plane. Finally, by solving the optimization problem, we obtain the optimal plane and perform color conversion. Simulation results show that the proposed algorithm outperforms the state-of-the-art algorithms in preserving both local details and naturalness ofimages.

A novel approach for image demosaicking based on adaptive lattice-aware filter (ALF) and global refinement unit (GRU) is proposed in this work. We generate ALFs dynamically, which are adaptive to positions of pixels within color lattices in a color filter array, to obtain a locally demosaicked image. We then refine the locally demosaicked image using GRU to exploit global information, as well as local information. To extend the receptive fields efficiently, we adopt dilated convolutions in GRU. Experimental results demonstrate that the proposed algorithm provides the state-of-the-art performances in standard demosaicking datasets.

An interactive image segmentation algorithm, which accepts user-annotations about a target object and the background, is proposed in this work. We convert user-annotations into interaction maps by measuring distances of each pixel to the annotated locations. Then, we perform the forward pass in a convolutional neural network, which outputs an initial segmentation map. However, the user-annotated locations can be mislabeled in the initial result. Therefore, we develop the backpropagating refinement scheme (BRS), which corrects the mislabeled pixels. Experimental results demonstrate that the proposed algorithm outperforms the conventional algorithms on four challenging datasets. Furthermore, we demonstrate the generality and applicability of BRS in other computer vision tasks, by transforming existing convolutional neural networks into user-interactive ones.

We propose a novel algorithm for monocular depth estimation using relative depth maps. First, using a convolutional neural network, we estimate relative depths between pairs of regions, as well as ordinary depths, at various scales. Second, we restore relative depth maps from selectively estimated data based on the rank-1 property of pairwise comparison matrices. Third, we decompose ordinary and relative depth maps into components and recombine them optimally to reconstruct a final depth map. Experimental results show that the proposed algorithm provides the state-of-art depth estimation performance.

A novel algorithm to estimate instance-level future motion in a single image is proposed in this paper. We first represent the future motion of an instance with its direction, speed, and action classes. Then, we develop a deep neural network that exploits different levels of semantic information to perform the future motion estimation. For effective future motion classification, we adopt ordinal regression. Especially, we develop the cyclic ordinal regression scheme using binary classifiers. Experiments demonstrate that the proposed algorithm provides reliable performance and thus can be used effectively for vision applications, including single and multi object tracking. Furthermore, we release the future motion (FM) dataset, collected from diverse sources and annotated manually, as a benchmark for single-image future motion estimation.

We propose a unified approach to three tasks of aesthetic score regression, binary aesthetic classification, and personalized aesthetics. First, we develop a comparator to estimate the ratio of aesthetic scores for two images. Then, we construct a pairwise comparison matrix for multiple reference images and an input image, and predict the aesthetic score of the input via the eigenvalue decomposition of the matrix. By varying the reference images, the proposed algorithm can be used for binary aesthetic classification and personalized aesthetics, as well as generic score regression. Experimental results demonstrate that the proposed unified algorithm provides the state-of-the-art performances in all three tasks of image aesthetics.

We propose a deep learning algorithm for single-image depth estimation based on the Fourier frequency domain analysis. First, we develop a convolutional neural network structure and propose a new loss function, called depth-balanced Euclidean loss, to train the network reliably for a wide range of depths. Then, we generate multiple depth map candidates by cropping input images with various cropping ratios. In general, a cropped image with a small ratio yields depth details more faithfully, while that with a large ratio provides the overall depth distribution more reliably. To take advantage of these complementary properties, we combine the multiple candidates in the frequency domain. Experimental results demonstrate that proposed algorithm provides the state-of-art performance. Furthermore, through the frequency domain analysis, we validate the efficacy of the proposed algorithm in most frequency bands.

Image aesthetic assessment is important for finding well taken and appealing photographs but is challenging due to the ambiguity and subjectivity of aesthetic criteria. We develop the pairwise aesthetic comparison network (PAC-Net), which consists of two parts: aesthetic feature extraction and pairwise feature comparison. To alleviate the ambiguity and subjectivity, we train PAC-Net to learn the relative aesthetic ranks of two images by employing a novel loss function, called aesthetic-adaptive cross entropy loss. Then, we develop simple schemes for using PAC-Net in the tasks of aesthetic ranking and aesthetic classification, respectively. Experimental results demonstrate that PAC-Net achieves the state-of-the-art performances in both the ranking and classification applications.

A reliable algorithm for rendering depth of field (DoF) effects using estimated depth maps, obtained through stereo matching, is proposed in this paper. The proposed algorithm generates blurring to simulate images spontaneously seen by human vision systems. We develop two types of windows : circle of confusion (CoC) blurring window and peripheral blurring window. First, the CoC blurring window is determined by comparing the depth values of a gazing point and each sample point. Second, the peripheral blurring window is obtained by calculating the distance between the gazing and sample points. Then, we combine the two windows to make the total blurring window. Finally, through a masking process, we modulate the total blurring window to provide a more natural DoF. Experimental results demonstrate that the proposed algorithm provides realistic blurring, by preserving edges clearly as well as blurring far points from the gazing point effectively.

A background subtraction algorithm using an encoder-decoder structured convolutional neural network is proposed in this work, in order to segment out moving objects from the background. A target frame, its previous frame, and a background model are concatenated and fed into the network as the input. Then, the encoder generates a highlevel feature vector, and the decoder converts the feature vector into a segmentation map, which roughly identifies moving object regions. Moreover, we develop background modeling and foreground extraction techniques, which exploit contour information. Experimental results on the CD-net2014 dataset demonstrate that the proposed algorithm outperforms state-of-the-art techniques significantly.

A semi-supervised online video object segmentation algorithm, which accepts user annotations about a target object at the first frame, is proposed in this work. We propagate the segmentation labels at the previous frame to the current frame using optical flow vectors. However, the propagation is error-prone. Therefore, we develop the convolutional trident network (CTN), which has three decoding branches: separative, definite foreground, and definite background decoders. Then, we perform Markov random field optimization based on outputs of the three decoders. We sequentially carry out these processes from the second to the last frames to extract a segment track of the target object. Experimental results demonstrate that the proposed algorithm significantly outperforms the state-of-the-art conventional algorithms on the DAVIS benchmark dataset.

A novel algorithm to segment a primary object in a video sequence is proposed in this work. First, we generate candidate regions for the primary object using both color and motion edges. Second, we estimate initial primary object regions, by exploiting the recurrence property of the primary object. Third, we augment the initial regions with missing parts or reducing them by excluding noisy parts repeatedly. This augmentation and reduction process (ARP) identifies the primary object region in each frame. Experimental results demonstrate that the proposed algorithm significantly outperforms the state-of-the-art conventional algorithms on recent benchmark datasets.

A novel contour-constrained superpixel (CCS) algorithm is proposed in this work. We initialize superpixels and regions in a regular grid and then refine the superpixel label of each region hierarchically from block to pixel levels. To make superpixel boundaries compatible with object contours, we propose the notion of contour pattern matching and formulate an objective function including the contour constraint. Furthermore, we extend the CCS algorithm to generate temporal superpixels for video processing. We initialize superpixel labels in each frame by transferring those in the previous frame and refine the labels to make superpixels temporally consistent as well as compatible with object contours. Experimental results demonstrate that the proposed algorithm provides better performance than the state-of-the-art superpixel methods.

A novel online algorithm to segment multiple objects in a video sequence is proposed in this work. We develop the collaborative detection, tracking, and segmentation (CDTS) technique to extract multiple segment tracks accurately. First, we jointly use object detector and tracker to generate multiple bounding box tracks for objects. Second, we transform each bounding box into a pixel-wise segment, by employing the alternate shrinking and expansion (ASE) segmentation. Third, we refine the segment tracks, by detecting object disappearance and reappearance cases and merging overlapping segment tracks. Experimental results show that the proposed algorithm significantly surpasses the state-of-the-art conventional algorithms on benchmark datasets.

A temporal superpixel algorithm based on proximity-weighted patch matching (TS-PPM) is proposed in this work. We develop the proximity-weighted patch matching (PPM), which estimates the motion vector of a superpixel robustly, by considering the patch matching distances of neighboring superpixels as well as the target superpixel. In each frame, we initialize superpixels by transferring the superpixel labels of the previous frame using PPM motion vectors. Then, we update the superpixel labels of boundary pixels, based on a cost function, composed of color, spatial, contour, and temporal consistency terms. Finally, we execute superpixel splitting, merging, and relabeling to regularize superpixel sizes and reduce incorrect labels. Experiments show that the proposed algorithm outperforms the state-of-the-art conventional algorithms significantly.

Semantic lines characterize the layout of an image. Despite their importance in image analysis and scene understanding, there is no reliable research for semantic line detection. In this paper, we propose a semantic line detector using a convolutional neural network with multi-task learning, by regarding the line detection as a combination of classification and regression tasks. We use convolution and max-pooling layers to obtain multi-scale feature maps for an input image. Then, we develop the line pooling layer to extract a feature vector for each candidate line from the feature maps. Next, we feed the feature vector into the parallel classification and regression layers. The classification layer decides whether the line candidate is semant ic or not. In case of a semantic line, the regression layer determines the offset for refining the line location. Experimental results show that the proposed detector extracts semantic lines accurately and reliably. Moreover, we demonstrate that the proposed detector can be used successfully in three applications: horizon estimation, composition enhancement, and image simplification.

We investigate the impacts of objective functions on the performance of deep-learning-based prostate magnetic resonance image segmentation. To this end, we first develop a baseline convolutional neural network (BCNN) for the prostate image segmentation, which consists of encoding, bridge, decoding, and classification modules. In the BCNN, we use 3D convolutional layers to consider volumetric information. Also, we adopt the residual feature forwarding and intermediate feature propagation techniques to make the BCNN reliably trainable for various objective functions. We compare six objective functions: Hamming distance, Euclidean distance, Jaccard index, dice coefficient, cosine similarity, and cross entropy. Experimental results on the PROMISE12 dataset demonstrate that the cosine similarity provides the best segmentation performance, whereas the cross entropy performs the worst.

A novel subpixel rendering algorithm for diamond-shaped PenTile displays, which improves the apparent display resolution and reduces color fringing artifacts, is proposed in this work. We develop two types of filters: main filters and nonnegative filters. First, the main filters are derived to minimize the error between an original input image and a perceived image. Second, the nonnegative filters are modified from the main filters. While the main filters are optimized for improving the resolution, the nonnegative filters suppress color distortions effectively. We analyze local characteristics of an image and estimate the amount of color distortions in each region. Based on the color distortion analysis, we render the image by applying adaptive combinations of the main filters and the nonnegative filters. Experimental results demonstrate that the proposed algorithm outperforms conventional algorithms, by not only improving the apparent resolution but also suppressing color distortions effectively.

The image quality of an in-vehicle black box camera is often degraded by the reflections of internal objects, dirt, and dust on the windshield. In this paper, we propose a novel algorithm that simultaneously removes the reflections and small dirt artifacts from in-vehicle black box videos under fast forward camera motion. The algorithm exploits the spatiotemporal coherence of the reflection and dirt, which remain stationary relative to the fast-moving background. Unlike previous algorithms, the algorithm first separates stationary reflection and then restores the background scene. To this end, we propose an average image prior, thereby imposing spatiotemporal coherence. The separation model is a two-layer model composed of stationary and background layers, where different gradient sparsity distributions are utilized in a region-based manner. Motion compensation in postprocessing is proposed to alleviate layer jitter due to vehicle vibrations. In evaluation experiments, the proposed algorithm successfully extracts the stationary layer from several real and synthetic black box videos.

In this paper, we propose a simple yet effective object descriptor and a novel tracking algorithm to track a target object accurately. For the object description, we divide the bounding box of a target object into multiple patches and describe them with color and gradient histograms. Then, we determine the foreground weight of each patch to alleviate the impacts of background information in the bounding box. To this end, we perform random walk with restart (RWR) simulation. We then concatenate the weighted patch descriptors to yield the spatially ordered and weighted patch (SOWP) descriptor. For the object tracking, we incorporate the proposed SOWP descriptor into a novel tracking algorithm, which has three components: locator, checker, and scaler (LCS). The locator and the scaler estimate the center location and the size of a target, respectively. The checker determines whether it is safe to adjust the target scale in a current frame. These three components cooperate with one another to achieve robust tracking. Experimental results demonstrate that the proposed LCS tracker achieves excellent performance on recent benchmarks.

A novel primary object discovery (POD) algorithm, which uses reliable object proposals while exploiting the recurrence property of a primary object in a video sequence, is proposed in this paper. First, we generate both color-based and motion-based object proposals in each frame, and extract the feature of each proposal using the random walk with restart simulation. Next, we estimate the foreground confidence for each proposal to remove unreliable proposals. By superposing the features of the remaining reliable proposals, we construct the primary object models. To this end, we develop the evolutionary primary object modeling technique, which exploits the recurrence property of the primary object. Then, using the primary object models, we choose the main proposal in each frame and find the location of the primary object by merging the main proposal with candidate proposals selectively. Finally, we refine the discovered bounding boxes by exploiting temporal correlations of the recurring primary object. Extensive experimental results demonstrate that the proposed POD algorithm significantly outperforms conventional algorithms.

An unsupervised video object segmentation algorithm, which discovers a primary object in a video sequence automatically, is proposed in this work. We introduce three energies in terms of foreground and background probability distributions: Markov, spatiotemporal, and antagonistic energies. Then, we minimize a hybrid of the three energies to separate a primary object from its background. However, the hybrid energy is nonconvex. Therefore, we develop the alternate convex optimization (ACO) scheme, which decomposes the nonconvex optimization into two quadratic programs. Moreover, we propose the forward-backward strategy, which performs the segmentation sequentially from the first to the last frames and then vice versa, to exploit temporal correlations. Experimental results on extensive datasets demonstrate that the proposed ACO algorithm outperforms the state-of-the-art techniques significantly.

A primary object discovery (POD) algorithm for a video sequence is proposed in this work, which is capable of discovering a primary object, as well as identifying noisy frames that do not contain the object. First, we generate object proposals for each frame. Then, we bisect each proposal into foreground and background regions, and extract features from each region. By superposing the foreground and background features, we build the object recurrence model, the background model, and the primary object model. We develop an iterative scheme to refine each model evolutionarily using the information in the other models. Finally, using the evolved primary object model, we select candidate proposals and locate the bounding box of a primary object by merging the proposals selectively. Experimental results on a challenging dataset demonstrate that the proposed POD algorithm extracts primary objects accurately and robustly.

An efficient stereo matching algorithm, which applies adaptive smoothness constraints using texture and edge information, is proposed in this work. First, we determine non-textured regions, on which an input image yields flat pixel values. In the non-textured regions, we penalize depth discontinuity and complement the primary CNN-based matching cost with a color-based cost. Second, by combining two edge maps from the input image and a pre-estimated disparity map, we extract denoised edges that correspond to depth discontinuity with high probabilities. Thus, near the denoised edges, we penalize small differences of neighboring disparities. Based on these adaptive smoothness constraints, the proposed algorithm outperforms the conventional methods significantly and achieves the state-of-the-art performance on the Middlebury stereo benchmark.

A novel RGB-D image segmentation algorithm is proposed in this work. This is the first attempt to achieve image segmentation based on the theory of multiple random walkers (MRW). We construct a multi-layer graph, whose nodes are superpixels divided with various parameters. Also, we set an edge weight to be proportional to the similarity of color and depth features between two adjacent nodes. Then, we segment an input RGB-D image by employing MRW simulation. Specifically, we decide the initial probability distribution of agents so that they are far from each other. We then execute the MRW process with the repulsive restarting rule, which makes the agents repel one another and occupy their own exclusive regions. Experimental results show that the proposed MRW image segmentation algorithm provides competitive segmentation performances, as compared with the conventional state-of-the-art algorithms.

A graph-based system to simulate the movements and interactions of multiple random walkers (MRW) is proposed in this work. In the MRW system, multiple agents traverse a single graph simultaneously. To achieve desired interactions among those agents, a restart rule can be designed, which determines the restart distribution of each agent according to the probability distributions of all agents. In particular, we develop the repulsive rule for data clustering. We illustrate that the MRW clustering can segment real images reliably. Furthermore, we propose a novel image cosegmentation algorithm based on the MRW clustering. Specifically, the proposed algorithm consists of two steps: inter-image concurrence computation and intra-image MRW clustering. Experimental results demonstrate that the proposed algorithm provides promising cosegmentation performance.

The notion of multihypothesis trajectory analysis (MTA) for robust visual tracking is proposed in this work. We employ multiple component trackers using texture, color, and illumination invariant features, respectively. Each component tracker traces a target object forwardly and then backwardly over a time interval. By analyzing the pair of the forward and backward trajectories, we measure the robustness of the component tracker. To this end, we extract the geometry similarity, the cyclic weight, and the appearance similarity from the forward and backward trajectories. We select the optimal component tracker to yield the maximum robustness score, and use its forward trajectory as the final tracking result. Experimental results show that the proposed MTA tracker improves the robustness and the accuracy of tracking, outperforming the state-of-the-art trackers on a recent benchmark dataset.

A simple yet effective object descriptor for visual tracking is proposed in this paper. We first decompose the bounding box of a target object into multiple patches, which are described by color and gradient histograms. Then, we concatenate the features of the spatially ordered patches to represent the object appearance. Moreover, to alleviate the impacts of background information possibly included in the bounding box, we determine patch weights using random walk with restart (RWR) simulations. The patch weights represent the importance of each patch in the description of foreground information, and are used to construct an object descriptor, called spatially ordered and weighted patch (SOWP) descriptor. We incorporate the proposed SOWP descriptor into the structured output tracking framework. Experimental results demonstrate that the proposed algorithm yields significantly better performance than the state-of-the-art trackers on a recent benchmark dataset, and also excels in another recent benchmark dataset.

A frame-level video matching algorithm, which achieves dense frame matching between near-duplicate videos, is proposed in this work. First, we propose a ternary frame descriptor for the near-duplicate video matching. The ternary descriptor partitions a frame into patches and uses ternary digits to represent relations between pairs of patches. Second, we formulate the frame-level matching problem as the minimization of a cost function, which consists of matching costs and adaptive unmatching costs. We develop an iterative refinement scheme that converges to a local minimum of the cost function. The iterative scheme performs competitively with the global optimization techniques while demands a significantly lower computational complexity. Experimental results show that the proposed algorithm achieves effective frame description and efficient frame matching of near duplicate videos.

A dark image enhancement algorithm based on the pairwise target contrast and the multi-scale detail boosting is proposed in this work. We first compute the pairwise target contrast between a pair of pixels, which represents the desired gray level difference of the two pixels in the output image. By aggregating the pairwise target contrasts for all pairs of pixels in an image, we formulate a cost function for the enhancement. By minimizing the cost function, we obtain the optimal transformation function and enhance the image. In addition, we propose a multi-scale approach to boost details in the globally enhanced image. Experimental results show that the proposed algorithm enhances the contrast and visibility of dark images more effectively than conventional algorithms.

A robust contrast enhancement algorithm for noisy low-light images, called the denoising-enhancement-completion (DEC), is proposed in this work. We observe that noise components in low-light images degrade the performance of the contrast enhancement. Therefore, we first reduce noise components in an input image. Then, we compute the reliability weight for each pixel, by measuring the difference between the input image and the denoised image, and categorize each pixel into one of two classes: noise-free or noisy. We perform the selective histogram equalization to enhance the contrast of the noise-free pixels only. Finally, we restore missing values of the noisy pixels using the enhanced noise-free pixel values, by employing a low-rank matrix completion scheme. Experimental results show that the proposed DEC algorithm removes noise and enhances the contrast of low-light images more effectively than conventional algorithms.

We propose a fast quality metric for depth maps, called fast depth quality metric (FDQM), which efficiently evaluates the impacts of depth map errors on the qualities of synthesized intermediate views in multiview video plus depth applications. In other words, the proposed FDQM assesses view synthesis distortions in the depth map domain, without performing the actual view synthesis. First, we estimate the distortions at pixel positions, which are specified by reference disparities and distorted disparities, respectively. Then, we integrate those pixel-wise distortions into an FDQM score by employing a spatial pooling scheme, which considers occlusion effects and the characteristics of human visual attention. As a benchmark of depth map quality assessment, we perform a subjective evaluation test for intermediate views, which are synthesized from compressed depth maps at various bitrates. We compare the subjective results with objective metric scores. Experimental results demonstrate that the proposed FDQM yields highly correlated scores to the subjective ones. Moreover, FDQM requires at least 10 times less computations than conventional quality metrics, since it does not perform the actual view synthesis.

A novel saliency detection algorithm for video sequences based on the random walk with restart (RWR) is proposed in this paper. We adopt RWR to detect spatially and temporally salient regions. More specifically, we first find a temporal saliency distribution using the features of motion distinctiveness, temporal consistency, and abrupt change. Among them, the motion distinctiveness is derived by comparing the motion profiles of image patches. Then, we employ the temporal saliency distribution as a restarting distribution of the random walker. In addition, we design the transition probability matrix for the walker using the spatial features of intensity, color, and compactness. Finally, we estimate the spatiotemporal saliency distribution by finding the steady-state distribution of the walker. The proposed algorithm detects foreground salient objects faithfully, while suppressing cluttered backgrounds effectively, by incorporating the spatial transition matrix and the temporal restarting distribution systematically. Experimental results on various video sequences demonstrate that the proposed algorithm outperforms conventional saliency detection algorithms qualitatively and quantitatively.

A novel algorithm to remove rain or snow streaks from a video sequence using temporal correlation and low-rank matrix completion is proposed in this paper. Based on the observation that rain streaks are too small and move too fast to affect the optical flow estimation between consecutive frames, we obtain an initial rain map by subtracting temporally warped frames from a current frame. Then, we decompose the initial rain map into basis vectors based on the sparse representation, and classify those basis vectors into rain streak ones and outliers with a support vector machine. We then refine the rain map by excluding the outliers. Finally, we remove the detected rain streaks by employing a low-rank matrix completion technique. Furthermore, we extend the proposed algorithm to stereo video deraining. Experimental results demonstrate that the proposed algorithm detects and removes rain or snow streaks efficiently, outperforming conventional algorithms.

We propose a video stabilization algorithm, which extracts a guaranteed number of reliable feature trajectories for robust mesh grid warping. We first estimate feature trajectories through a video sequence and transform the feature positions into rolling-free smoothed positions. When the number of the estimated trajectories is insufficient, we generate virtual trajectories by augmenting incomplete trajectories using a low-rank matrix completion scheme. Next, we detect feature points on a large moving object and exclude them so as to stabilize camera movements, rather than object movements. With the selected feature points, we set a mesh grid on each frame and warp each grid cell by moving the original feature positions to the smoothed ones. For robust warping, we formulate a cost function based on the reliability weights of each feature point and each grid cell. The cost function consists of a data term, a structure-preserving term, and a regularization term. By minimizing the cost function, we determine the robust mesh grid warping and achieve the stabilization. Experimental results demonstrate that the proposed algorithm reconstructs videos more stably than the conventional algorithms.

A novel visual tracking algorithm using patch-based appearance models is proposed in this paper. We first divide the bounding box of a target object into multiple patches and then select only pertinent patches, which occur repeatedly near the center of the bounding box, to construct the foreground appearance model. We also divide the input image into non-overlapping blocks, construct a background model at each block location, and integrate these background models for tracking. Using the appearance models, we obtain an accurate foreground probability map. Finally, we estimate the optimal object position by maximizing the likelihood, which is obtained by convolving the foreground probability map with the pertinence mask. Experimental results demonstrate that the proposed algorithm outperforms state-of-the-art tracking algorithms significantly in terms of center position errors and success rates.

An accurate quality metric, called GEQM, for gray-level edge maps based on the structural matching of edge pixels is proposed in this work. We design the positional matching cost, which reflects the distance between two edge pixels, and the structural matching cost, which measures the structural shapes of edges as well as the differences of edge strength levels. Based on the cost functions, we perform the graph-cut optimization to obtain the optimal pixel-based matching between source and target edge maps bidirectionally. Finally, we compute the GEQM score by summing up the optimal matching costs of all edge pixels. Experimental results show that the proposed GEQM performs the edge map quality assessment more accurately and more reliably than conventional metrics. Especially, GEQM is suitable for assessing the qualities of synthesized intermediate views in multi-view image processing.

A novel rain (or snow) streak removal algorithm for stereo video sequences is proposed in this work. We observe that rain streaks appear at different locations in spatiotemporally adjacent frames. Thus, to derain a left-view frame, we synthesize it by warping the spatially adjacent right-view frame and the temporally previous and next frames, respectively. We subtract each warped frame from the original frame, and apply the median filter to the three difference images to obtain a reliable rain mask. Then, we remove rain streaks by replacing each rainy pixel value with a weighted average of non-locally neighboring pixel values. Experimental results demonstrate that the proposed algorithm removes rain streaks reliably and recovers original scene contents faithfully.

A robust video stabilization algorithm, which reduces shaky camera movements and rolling-shutter distortions in video sequences, is proposed in this work. We first extract feature trajectories through a video sequence, and then transform the feature positions into rolling-free smoothed positions. Then, we set a mesh grid on each frame and warp each grid cell by matching the original features to the smoothed ones. For robust warping, we formulate a cost function based on the confidence of each feature and the reliability of each grid cell. The cost function consists of a data term, a structure-preserving term, and a regularization term. By minimizing the cost function, we find the optimal grid positions in the warped frame and transform each grid cell accordingly. Experimental results show that the proposed algorithm stabilizes videos and removes rolling-shutter distortions more efficiently than conventional algorithms.

A video saliency detection algorithm based on feature learning, called ROCT, is proposed in this work. To detect salient regions, we design multiple spatiotemporal features and combine those features using a support vector machine (SVM). We extract the spatial features of rarity, compactness, and center prior by analyzing the color distribution in each image frame. Also, we obtain the temporal features of motion intensity and motion contrast to identify visually important motions. We train an SVM classifier using the spatiotemporal features extracted from training video sequences. Finally, we compute the visual saliency of each patch in an input sequence using the trained classifier. Experimental results demonstrate that the proposed algorithm provides more accurate and reliable results of saliency detection than conventional algorithms.

A novel contrast enhancement (CE) algorithm using 2-dimensional (2D) histograms, which transforms pixel values adaptively based on the depth information, is proposed in this work. In general, foreground objects convey more important visual information than background regions. Hence we assign high CE priorities to foreground pixels using the depth values and generate a depth-guided 2D histogram. Then, we stretch the gray-level differences of adjacent foreground pixels more strongly than those of adjacent background pixels. Moreover, to enhance background regions as well, we design two transformation functions for the foreground and the background separately. By combining the two functions according to pixel depths, we obtain an adaptive space-variant transformation function, which is finally used to reconstruct the output image. Experimental results show that the proposed algorithm outperforms conventional CE algorithms by enhancing salient foreground objects efficiently and preserving background details faithfully.

A video genre classification algorithm based on the voting from multiple SVMs is proposed in this work. While conventional genre classifiers use generic baseline features, we employ more specialized features to describe five video genres: animation, commercial, entertainment, drama, and sports. We also present a robust classification algorithm using multiple SVMs, which consider all possible binary grouping of the five genres. Given a query video, each SVM casts a probabilistic vote for each genre. Then, the optimal genre with the maximum votes is selected. Experimental results show that the proposed algorithm provides more accurate classification performance than conventional algorithms.

In this paper, we propose a graph-based multiscale saliency-detection algorithm by modeling eye movements as a random walk on a graph. The proposed algorithm first extracts intensity, color, and compactness features from an input image. It then constructs a fully connected graph by employing image blocks as the nodes. It assigns a high edge weight if the two connected nodes have dissimilar intensity and color features and if the ending node is more compact than the starting node. Then, the proposed algorithm computes the stationary distribution of the Markov chain on the graph as the saliency map. However, the performance of the saliency detection depends on the relative block size in an image. To provide a more reliable saliency map, we develop a coarse-to-fine refinement technique for multiscale saliency maps based on the random walk with restart (RWR). Specifically, we use the saliency map at a coarse scale as the restarting distribution of RWR at a fine scale. Experimental results demonstrate that the proposed algorithm detects visual saliency precisely and reliably. Moreover, the proposed algorithm can be efficiently used in the applications of proto-object extraction and image retargeting.

An optimized brightness-compensated contrast enhancement (BCCE) algorithm for transmissive liquid crystal displays (LCDs) is proposed in this paper. We first develop a global contrast enhancement scheme to compensate for the reduced brightness when the backlight of an LCD device is dimmed for power reduction. We also derive a distortion model to describe the information loss due to the brightness compensation. Then, we formulate an objective function that consists of the contrast enhancement term and the distortion term. By minimizing the objective function, we maximize the backlight-scaled image contrast, subject to the constraint on the distortion. Simulation results show that the proposed BCCE algorithm provides high-quality images, even when the backlight intensity is reduced by up to 50-70% to save power.

An efficient bit allocation algorithm based on a novel view synthesis distortion model is proposed for the rate-distortion optimized coding of multiview video plus depth sequences in this paper. We decompose an input frame into nonedge blocks and edge blocks. For each nonedge block, we linearly approximate its texture and disparity values, and derive a view synthesis distortion model, which quantifies the impacts of the texture and depth distortions on the qualities of synthesized virtual views. On the other hand, for each edge block, we use its texture and disparity gradients for the distortion model. In addition, we formulate a bit-rate allocation problem in terms of the quantization parameters for texture and depth data. By solving the problem, we can optimally divide a limited bit budget between the texture and depth data, in order to maximize the qualities of synthesized virtual views, as well as those of encoded real views. Experimental results demonstrate that the proposed algorithm yields the average PSNR gains of 1.98 and 2.04 dB in two-view and three-view scenarios, respectively, as compared with a benchmark conventional algorithm.

A novel approach to adapt the macroblock (MB) coding order of chrominance planes is proposed to improve the compression performance of the H.264/AVC intra coder. The proposed algorithm first encodes the luminance plane of an image and then uses the luminance data to determine the MB coding order of the chrominance planes adaptively. The proposed algorithm consists of three steps. First, we estimate the similarities between neighboring chrominance MBs using the luminance gradients. Second, we assign a high priority to a chrominance MB, if it is similar to many neighboring MBs. Finally, we encode the chrominance MBs in the decreasing order of the priorities. Moreover, we develop efficient submodes, representing prediction directions, to exploit the correlations of the MBs along the adaptive scan order. Experimental results show that the proposed algorithm yields significantly better compression performance than H.264/AVC on both YUV 4:4:4 and RGB 4:4:4 images.

We propose a robust stereo matching algorithm for images captured under varying radiometric conditions, such as exposure and lighting variations, based on the cumulative distributions of gradients. The gradient operator extracts local changes in pixel values, which are less sensitive to radiometric variations than the original pixel values. Moreover, the cumulative distribution function (CDF) of gradient vectors reflects the ranks of edge strength levels, and corresponding pixels in stereo images tend to have similar ranks regardless of radio-metric conditions. Therefore, we design the matching cost function based on the dissimilarity of gradient CDF values. However, since multiple pixels in an image may have the same gradient CDF value, we further constrain the correspondence matching by checking the dissimilarity of gradient orientations. Finally, to estimate an accurate disparity at each pixel, we adaptively aggregate matching costs using the color similarity and the geometric proximity of neighboring pixels. Experimental results demonstrate that the proposed algorithm provides more accurate disparities than conventional algorithms, especially under varying lighting conditions.

A graph-based video saliency detection algorithm is proposed in this work. We model eye movements on an image plane as random walks on a graph. To detect the saliency of the first frame in a video sequence, we construct a fully connected graph, in which each node represents an image block. We assign an edge weight to be proportional to the dissimilarity between the incident nodes and inversely proportional to their geometrical distance. We extract the saliency level of each node from the stationary distribution of the random walker on the graph. Next, to detect the saliency of each subsequent frame, we add the criterion that an edge, connecting a slow motion node to a fast motion node, should have a large weight. We then compute the stationary distribution of the random walk with restart (RWR) simulation, in which the saliency of the previous frame is used as the restarting distribution. Experimental results show that the proposed algorithm provides more reliable and accurate saliency detection performance than conventional algorithms.

An adaptive rain streak removal algorithm for a single image is proposed in this work. We observe that a typical rain streak has an elongated elliptical shape with a vertical orientation. Thus, we first detect rain streak regions by analyzing the rotation angle and the aspect ratio of the elliptical kernel at each pixel location. We then perform the nonlocal means filtering on the detected rain streak regions by selecting nonlocal neighbor pixels and their weights adaptively. Experimental results demonstrate that the proposed algorithm removes rain streaks more efficiently and provides higher restored image qualities than conventional algorithms.

A novel probabilistic depth-guided multi-view denoising (PDMD) algorithm is proposed in this work. We formulate the multi-view image denoising problem by considering the uncertainties in depth estimates in noisy environments. Specifically, we employ the geometric distributions of nonlocal neighbors, as well as the block similarities, to approximate the probabilities of depth estimates. We then use those probabilities to average all nonlocal neighbors and perform the minimum mean square error (MMSE) denoising. Simulation results show that the proposed PDMD algorithm provides better denoising performance than conventional algorithms.

A fast visual object tracking algorithm using novel object appearance models is proposed in this work. We develop a color histogram model and a patch difference model to extract color and texture feature vectors, respectively. Then, we apply k-nearest neighbor classifiers to the color and texture feature vectors and obtain the foreground probability map. We then perform a hierarchical mean shift process on the map to identify the object window. Experimental results demonstrate that proposed algorithm outperforms the conventional algorithms in terms of both tracking accuracy and processing speed.

A robust optical flow estimation algorithm for motion-blurred regions is proposed in this work. We first obtain initial optical flow vectors. Then, we detect motion-blurred regions that yield low contrast, low saturation, and inconsistent optical flow vectors. We replace the optical flow vectors at motion-blurred pixels with reliable vectors at nearby unblurred pixels. To this end, we develop an energy minimization framework. Simulation results demonstrate that the proposed algorithm refines optical flow vectors in motion-blurred regions accurately and provides better performance than conventional algorithms.

A novel motion-compensated frame interpolation (MCFI) algorithm to increase video temporal resolutions based on multihypothesis motion estimation and texture optimization is proposed in this paper. Initially, we form multiple motion hypotheses for each pixel by employing different motion estimation parameters, i.e., different block sizes and directions. Then, we determine the best motion hypothesis for each pixel by solving a labeling problem and optimizing the parameters. In the labeling problem, the cost function is composed of color, shape, and smoothness terms. Finally, we refine the motion hypothesis field based on the texture optimization technique and blend multiple source pixels to interpolate each pixel in the intermediate frame. Simulation results demonstrate that the proposed algorithm provides significantly better MCFI performance than conventional algorithms.

A novel contrast enhancement algorithm based on the layered difference representation of 2D histograms is proposed in this paper. We attempt to enhance image contrast by amplifying the gray-level differences between adjacent pixels. To this end, we obtain the 2D histogram h(k, k + l ) from an input image, which counts the pairs of adjacent pixels with gray-levels k and k + l , and represent the gray-level differences in a tree-like layered structure. Then, we formulate a constrained optimization problem based on the observation that the gray-level differences, occurring more frequently in the input image, should be more emphasized in the output image. We first solve the optimization problem to derive the transformation function at each layer. We then combine the transformation functions at all layers into the unified transformation function, which is used to map input gray-levels to output gray-levels. Experimental results demonstrate that the proposed algorithm enhances images efficiently in terms of both objective quality and subjective quality.

An efficient fine-granular scalable coding algorithm of 3-D mesh sequences for low-latency streaming applications is proposed in this work. First, we decompose a mesh sequence into spatial and temporal layers to support scalable decoding. To support the finest-granular spatial scalability, we decimate only a single vertex at each layer to obtain the next layer. Then, we predict the coordinates of decimated vertices spatially and temporally based on a hierarchical prediction structure. Last, we quantize and transmit the spatio-temporal prediction residuals using an arithmetic coder. We propose an efficient context model for the arithmetic coding. Experiment results show that the proposed algorithm provides significantly better compression performance than the conventional algorithms, while supporting finer-granular spatial scalability.

A consistent stereo matching (CSM) algorithm under varying radiometric conditions, such as lighting and exposure variations, for intermediate view synthesis is proposed in this work. First, we transform the colors of stereo images adaptively so that they are similar at corresponding pixels. Since the correspondences are generally unknown before stereo matching, we estimate pseudo-disparity vectors by sorting pixels based on the cumulative color histograms and use those pseudo vectors in the color transform. Then, to improve the accuracy of stereo matching, we jointly estimate the disparity maps for virtual intermediate views as well as those for real views, based on the consistency criterion that an object point should have the same disparity through all the views. Specifically, we compute matching costs using the reliability term and aggregate the costs to obtain initial disparity maps. We then refine the initial disparity maps by minimizing an energy function, which includes the consistency term. Experimental results show that the proposed CSM algorithm significantly reduces the error rate of disparity estimation under different radiometric conditions and synthesizes high quality intermediate views.

We propose a correspondence matching algorithm for multi-view video sequences, which provides reliable performance even when the multiple cameras have significantly different parameters, such as viewing angles and positions. We use an activity vector, which represents the temporal occurrence pattern of moving foreground objects at a pixel position, as an invariant feature for correspondence matching. We first devise a novel similarity measure between activity vectors by considering the joint and individual behavior of the activity vectors. Specifically, we define random variables associated with the activity vectors and measure their similarity using the mutual information between the random variables. Moreover, to find a reliable homography transform between views, we find consistent pixel positions by employing the iterative bidirectional matching. We also refine the matching results of multiple source pixel positions by minimizing a matching cost function based on the Markov random field. Experimental results show that the proposed algorithm provides more accurate and reliable matching performance than the conventional activity-based and feature-based matching algorithms, and therefore can facilitate various applications of visual sensor networks.

A novel motion-compensated frame interpolation algorithm to increase video frame rates is proposed in this work, which employs texture optimization techniques to refine inaccurate motion vector fields. We enforce the congruent segment constraint in the motion refinement so that matching objects have similar local image structures. More specifically, we use the congruence energy as well as the appearance energy in the motion optimization to estimate high quality motion vectors. Simulation results show that, by preserving complex object shapes and texture, the proposed algorithm provides more faithful intermediate frames than conventional algorithms.

A real-time video dehazing algorithm, which reduces flickering artifacts and yields high quality output videos, is proposed in this work. Assuming that a scene point yields highly correlated transmission values between adjacent image frames, we develop the temporal coherence cost. Then, we add the temporal coherence cost to the contrast cost and the truncation loss cost to define the overall cost function. By minimizing the overall cost function, we obtain the optimal transmission. Moreover, to reduce the computational complexity and facilitate real-time applications, we approximate the conventional edge preserving filter by the overlapped block filter. Experimental results demonstrate that the proposed algorithm is sufficiently fast for real-time applications and effectively removes haze and flickering artifacts.

We propose a novel backlight-scaled contrast enhancement algorithm to maintain the image quality when the backlight of a TFT-LCD display is dimmed for power reduction. First, we derive the transformation function to maintain the perceived luminance and enhance the contrast. Second, we model the perceptual distortion, which describes the information loss due to the backlight scaling. Then, we propose a Lagrangian method to maximize the backlight-scaled image contrast subject to the constraint on the perceptual distortion. Simulation results show that the proposed algorithm provides better image qualities than the conventional algorithms.

A novel contrast enhancement algorithm based on the layered difference representation is proposed in this work. We first represent gray-level differences at multiple layers in a tree-like structure. Then, based on the observation that gray-level differences, occurring more frequently in the input image, should be more emphasized in the output image, we solve a constrained optimization problem to derive the transformation function at each layer. Finally, we aggregate the transformation functions at all layers into the overall transformation function. Simulation results demonstrate that the proposed algorithm enhances images efficiently in terms of both objective quality and subjective quality.

We propose a colorization-based prediction algorithm, which reduces the correlation between luminance and chrominance planes, to achieve compact compression of color images. We first develop a priority-based colorization scheme, which adds chrominance values to gray pixels using a set of colored pixels, assuming that neighboring pixels with similar luminance values tend to have similar chrominance values as well. The colorization scheme assigns a high priority to a pixel that has a large number of neighbors with similar luminance values. Then, we employ the colorization scheme to predict the chrominance planes from the luminance plane. Since the priority-based colorization scheme predicts color data accurately even in complicated color regions, the prediction residuals can be efficiently compressed. Simulation results on rich color images show that the proposed algorithm yields better compression performance than H.264/AVC.

A side information generation method using assistant pixels is proposed for distributed video coding (DVC) in this work. To generate higher quality side information frames and improve the rate-distortion (R-D) performance of a DVC system, we propose transmitting assistant pixels, which help to improve the motion estimation accuracy of the decoder at the cost of an overhead bit-rate. Also, we develop a mode decision scheme, which determines the interpolation mode for each block by combining forward and backward motion vectors. Simulation results demonstrate that the proposed algorithm provides better R-D performance than the state-of-the-art DVC algorithm.

An efficient coding algorithm for depth map images and videos, based on view synthesis distortion estimation, is proposed in this work. We first analyze how a depth error is related to a disparity error and how the disparity vector error affects the energy spectral density of a synthesized color video in the frequency domain. Based on the analysis, we propose an estimation technique to predict the view synthesis distortion without requiring the actual synthesis of intermediate view frames. To encode the depth information efficiently, we employ a Lagrangian cost function to minimize the view synthesis distortion subject to the constraint on a transmission bit rate. In addition, we develop a quantization scheme for residual depth data, which adaptively assigns bits according to block complexities. Simulation results demonstrate that the proposed depth video coding algorithm provides significantly better R-D performance than conventional algorithms.

A novel quality metric for binary edge maps, called the structural edge quality metric (SEQM), is proposed in this work. First, we define the matching cost between an edge pixel in a detected edge map and its candidate matching pixel in the ground-truth edge map. The matching cost includes a structural term, as well as a positional term, to measure the discrepancy between the local structures around the two pixels. Then, we determine the optimal matching pairs of pixels using the graph-cut optimization, in which a smoothness term is employed to take into account global edge structures in the matching. Finally, we sum up the matching costs of all edge pixels to determine the quality index of the detected edge map. Simulation results demonstrate that the proposed SEQM provides more faithful and reliable quality indices than conventional metrics.

A histogram-based matching algorithm for stereo images captured under different illumination conditions is proposed in this work. The cumulative histogram of an image represents the ranks of relative pixel brightness, which are robust to illumination changes. Therefore, we design the matching cost based on the similarity of the cumulative histograms of stereo images. As an optional mode, the proposed algorithm can evaluate the histograms for foreground objects and the background separately to alleviate occlusion artifacts. To determine the disparity of each pixel, the proposed algorithm adaptively aggregates matching costs based on the color similarity and the geometric proximity of neighboring pixels. Then, it refines false disparities at occluded pixels using more reliable disparities of non-occluded pixels. Experimental results demonstrate that the proposed algorithm provides higher quality disparity maps than the conventional methods under varying illumination conditions.

A power-constrained contrast-enhancement algorithm for emissive displays based on histogram equalization (HE) is proposed in this paper. We first propose a log-based histogram modification scheme to reduce overstretching artifacts of the conventional HE technique. Then, we develop a power-consumption model for emissive displays and formulate an objective function that consists of the histogram-equalizing term and the power term. By minimizing the objective function based on the convex optimization theory, the proposed algorithm achieves contrast enhancement and power saving simultaneously. Moreover, we extend the proposed algorithm to enhance video sequences, as well as still images. Simulation results demonstrate that the proposed algorithm can reduce power consumption significantly while improving image contrast and perceptual quality.

A simple and adaptive single image dehazing algorithm is proposed in this work. Based on the observation that a hazy image has low contrast in general, we attempt to restore the original image by enhancing the contrast. First, the proposed algorithm estimates the airlight in a given hazy image based on the quad-tree subdivision. Then, the proposed algorithm estimates the transmission map to maximize the contrast of the output image. To measure the contrast, we develop a cost function, which consists of a standard deviation term and a histogram uniformness term. Experimental results demonstrate that the proposed algorithm can remove haze efficiently and reconstruct fine details in original scenes clearly.

A novel algorithm for thumbnail generation, which preserves characteristic features of a source image including blurs and textures, is proposed in this work. When a source image is subsampled to generate a thumbnail, important visible cues, such as blurs and noises, are lost. To overcome this drawback, we first create multiple thumbnail candidates that accentuate three classes of image features: focal blur, motion blur, and detail. Then, we obtain the final thumbnail by composing these candidates adaptively. Assuming that image features are spatially varying but locally static, we formulate the composition task as a labeling problem, and employ the graph-cut optimization technique to solve the problem. Simulation results demonstrate that the proposed algorithm provides feature-preserving thumbnails efficiently.

A three-dimensional (3D) mesh compression algorithm based on novel prediction methods and a mode decision scheme is proposed in this work. After decomposing an input mesh into base and refinement layers, we segment the geometry data of each layer into clusters. To encode vertex positions efficiently, we propose two prediction methods: the dual ring prediction and the minimum mean square error (MMSE) prediction. Also, we develop a mode decision scheme that selects the best prediction mode for each cluster. Simulation results demonstrate that the proposed algorithm provides significantly better compression performance than conventional techniques.

A nonlocal minimum mean square error (MMSE) image denoising algorithm to remove Poisson noise is proposed in this work. Based on the Bayesian estimation theory, we first derive the nonlocal MMSE denoising filter, which can minimize the mean square error (MSE) of a denoised block. Then, we develop an approximation of the filter for practical implementation. Simulation results show that the proposed algorithm provides significantly better denoising performance than the conventional nonlocal means filter and its recent extension for Poisson noise.

A novel contrast enhancement algorithm with histogram-guided boundary conditions is proposed in this work. The proposed algorithm enhances details in local regions by boosting gradient components, while improving the overall contrast by imposing boundary conditions based on a global transformation function. Moreover, we develop an efficient masking scheme, called the soft masking, to strike the balance between the global enhancement and the local enhancement. Simulation results demonstrate that the proposed algorithm can yield high quality output images by improving both global and local contrast simultaneously.

In this work, we propose a progressive compression algorithm using topology-based Karhunen-Loeve transform(KLT). First, we simplify an input mesh to represents an original mesh in several level of details. Then, coordinates of decimated vertices at each level are predicted from the coarser level mesh, and the prediction residuals are transmitted to a decoder. To provide high coding efficiency, we apply the topology-based KLT, which compacts the energy into a few coefficients, to the prediction residuals. Moreover, we develop a bit plane coder, which uses a context-adaptive arithmetic coder, for the entropy coding. Experiments on various 3D meshes show that the proposed algorithm provides enhanced compression performance.

An efficient frame loss concealment algorithm for stereoscopic video based on the inter-view similarity of motion vectors and intensity differences is proposed in this work. Suppose that a frame at time t in the right view is lost during the transmission. To conceal its loss, we use the information in the previous frame at time t - 1 in the right view and the frames at t - 1 and t in the left view. More specifically, we first estimate the disparity vector field of the previous frame to find the matching pixels in the left view, and determine the motion vectors and the intensity differences of those matching pixels. By projecting those motion vectors and intensity differences onto the right view, we recover the lost frame. Simulation results demonstrate that the proposed algorithm provides significantly better performance than conventional algorithms.

An algorithm for compensating the effects of backlight in images and videos is proposed in this work. We first determine the region of interest (ROI) to compensate mainly using a saliency map, which is based on darkness, skin color, color and texture prominency features. We then compute the compensating offset value for each pixel. Initial offset values are derived to improve the brightness and the contrast of the ROI, and also to provide temporally consistent output frames in case of the video compensation. Finally, we obtain the final offset values by minimizing an energy function, consisting of a data term and a smoothness term. Simulation results show that the proposed algorithm improves the picture qualities of backlit images and videos efficiently.

A novel power-constrained contrast enhancement algorithm for organic light-emitting diode (OLED) displays is proposed in this work. We first develop the log-modified histogram equalization (LMHE) scheme, which reduces overstretching artifacts of the conventional histogram equalization technique. Then, we model the power consumption in OLED displays, and incorporate it into LMHE to achieve the optimal tradeoff between contrast enhancement and power saving. Simulation results demonstrate that the proposed algorithm can reduce the power consumption significantly, while preserving image qualities.

An algorithm to generate a panorama from multi-view images, which contain foreground objects with varying depths, is proposed in this work. The proposed algorithm constructs a foreground panorama and a background panorama separately, and then merges them into a complete panorama. First, the foreground panorama is obtained by finding the translational displacements of objects between source images. Second, the background panorama is initialized using warped source images and then optimized to preserve spatial consistency and satisfy visual constraints. Then, the background panorama is extended by inserting seams and merged with the foreground panorama. Experimental results demonstrate that the proposed algorithm provides visually satisfying panoramas with all meaningful foreground objects, but without severe artifacts in the backgrounds.

The coding gain that can be achieved by improving the coding order of B frames in the H.264/AVC standard is investigated in this work. We first represent the coding order of B frames and their reference frames with a binary tree. We then formulate a recursive equation to find out the binary tree that provides a suboptimal, but very efficient, coding order. The recursive equation is efficiently solved using a dynamic programming method. Furthermore, we extend the coding order improvement technique to the case of multiview video sequences, in which the quadtree representation is used instead of the binary tree representation. Simulation results demonstrate that the proposed algorithm provides significantly better R-D performance than conventional prediction structures.

An adaptive image and video retargeting algorithm based on Fourier analysis is proposed in this work. We first divide an input image into several strips using the gradient information so that each strip consists of textures of similar complexities. Then, we scale each strip adaptively according to its importance measure. More specifically, the distortions, generated by the scaling procedure, are formulated in the frequency domain using the Fourier transform. Then, the objective is to determine the sizes of scaled strips to minimize the sum of distortions, subject to the constraint that the sum of their sizes should equal the size of the target output image. We solve this constrained optimization problem using the Lagrangian multiplier technique. Moreover, we extend the approach to the retargeting of video sequences. Simulation results demonstrate that the proposed algorithm provides reliable retargeting performance efficiently.

An efficient colorization scheme for images and videos based on prioritized source propagation is proposed in this work. A user first scribbles colors on a set of source pixels in an image or the first frame of a movie. The proposed algorithm then propagates those colors to the other non-source pixels and the subsequent frames. Specifically, the proposed algorithm identifies the non-source pixel with the highest priority, which can be most reliably colorized. Then, its color is interpolated from the neighboring pixels. This is repeated until the whole image or movie is colorized. Simulation results demonstrate that the proposed algorithm yields more reliable colorization performance than the conventional algorithms.

A virtual view synthesis algorithm using multi-view video sequences, which inherits the advantages of both the forward warping and the inverse warping, is proposed in this work. First, we use the inverse warping to synthesize a virtual view without holes from the nearest two views. Second, we detect occluded regions in the synthesized view based on the uniqueness constraint. Then, we refine the occluded regions using the information in the farther views based on the forward warping technique. Simulation results demonstrate that the proposed algorithm provides significantly higher PSNR performances than the conventional inverse warping scheme.

An efficient background subtraction algorithm using two-level sampling and silhouette detection is proposed in this work. In the two-level sampling, we identify moving objects at the block level and then at the pixel level. Then, in the silhouette detection, around each sampled foreground pixel, we refine the shapes of foreground objects. We also develop two fast modes for the silhouette detection, which utilizes the spatio-temporal coherence of moving foreground objects. Simulation results demonstrate that the proposed algorithm provides accurate segmentation results without flickering artifacts, while requiring a low computational load.

A novel concept of channel division to improve the performance of distributed video coding is proposed in this work. At the decoder, the proposed algorithm partitions each side information frame into multiple regions with different expected distortions. It is shown that the partitioning is equivalent to the division of the virtual noisy channel from the encoder to the decoder into multiple channels. Then, the proposed algorithm analyzes the noise characteristics of those multiple channels, and allocates a limited bit budget to those channels adaptively to improve the rate-distortion performance. Simulation results demonstrate that the proposed algorithm provides up to 5 dB better performance than the conventional DVC algorithms, as well as reduces the decoding complexity significantly.

We propose a novel video codec that can distribute computational complexity between the encoder and the decoder in a flexible manner. Since motion estimation is the most computationally intensive part in video coding, it is important to share the motion estimation task. In the proposed algorithm, the decoder estimates motion vectors by performing a partial three step search. The estimated motion vectors are sent back to the encoder via a feedback channel. The encoder then refines the received motion vectors. In this work, four operation modes corresponding to the encoder to decoder complexity ratios are presented. Experimental results demonstrate that the proposed algorithm allocates the complexity between the encoder and the decoder effectively, while providing promising compression performance.

We propose two temporal feature modulation algorithms that extract a feature from each video frame and modulate the features for a series of frames to embed a watermark codeword. In the first algorithm, the existence of a frame is used as the frame feature, and a watermark codeword is embedded into the original video by skipping selected frames. In the second algorithm, the centers of gravity of blocks in a frame are used as the frame feature. By modifying the centers of gravity, we embed 1-bit information into the frame. Simulation results demonstrate that the proposed algorithms are robust against compression and temporal attacks.

A real-time video segmentation algorithm, which can extract objects from video sequences even with non-stationary backgrounds, is proposed in this work. First, we segment the first frame into an object and a background interactively to build the probability density functions of colors in the object and the background. Then, for each subsequent frame, we construct a coherence strip, which is likely to contain the object contour, by exploiting spatio-temporal correlations. Finally, we perform the segmentation by minimizing an energy function composed of color, coherence, and smoothness terms. Experimental results on various test sequences show that the proposed algorithm provides accurate segmentation results in real-time, even though video sequences contain unstable camera motions.

In this work, we propose an efficient coding algorithm for multi-view video sequences, captured with a 2-D camera array. We develop an efficient prediction structure, which can exploit inter-view redundancies as well as temporal redundancies in 2-D multi-view video sequences. Moreover, we propose a view interpolation technique based on the bilateral criterion and the skip mode to improve the coding performance. Simulation results show that the proposed algorithm provides better PSNR performance than the conventional multi-view video coding algorithm.

An inpainting algorithm for multi-view video sequences is proposed in this work. First, we set a mask region in a view with manual interactions, and find the corresponding masks in the other views using the mask cloning method. Then, we overlay grid points on the mask region in the target view frame. For each grid rectangle, we find the matching patch in the source view frame. Finally, we transform and paste the patches to the target frame. Simulation results illustrate that the proposed algorithm removes objects in the multi-view sequences effectively by exploiting inter-view redundancies.

In ubiquitous network, node location information is the most essential data for efficiently managing increased mobility and supporting location based services. Therefore, we propose a next generation location information based network, which uses address containing location information. Finally, its functions, effectiveness, and feasibility will be verified through this demonstration.

An image enhancement algorithm based on the sorted histogram equalization (SHE) and specification (SHS) is proposed in this work. Although SHE and SHS can generate an arbitrary output histogram exactly, they yield false contour artifacts and amplify noises in many cases. To reduce these artifacts, we propose a postprocessing scheme using the projection onto convex sets (POCS) theory. Specifically, the proposed algorithm iteratively processes the output images of SHE and SHS using the low-pass, boundary and dithering conditions. Simulation results demonstrate that the proposed algorithm provides an excellent image quality, by improving the contrast while reducing undesired artifacts.

An algorithm to construct the optimal prediction structure in multiview video coding (MVC) is proposed in this work. We employ the graph theory as a framework. By considering each frame as a vertex and the motion compensation or disparity compensation as an edge, we represent a prediction structure as a spanning tree. Then, we obtain the optimal structure by finding the minimum spanning tree using the Prim's algorithm. Simulation results demonstrate that the proposed algorithm provides about 0.2-0.4 dB better PSNR performance than the conventional prediction structure, and about 1.5 dB better performance than the simulcast.

A gradient domain tone mapping algorithm is proposed to display high dynamic range (HDR) video sequences in low dynamic range (LDR) devices in this work. The proposed algorithm obtains a pixelwise motion vector field and incorporates the motion information into the Poisson equation. Then, by attenuating large spatial gradients, the proposed algorithm can yield a high-quality tone-mapped result without flickering artifacts. Simulation results show that the proposed algorithm provides a better performance than the frame-based method, which processes each frame independently.

We present a multiple description coding (MDC) scheme for 3D plane-based surfaces. First, planes are split into two disjoint subsets, called descriptions, each of which provides an equal contribution in 3D surface reconstruction. To optimize the quality of the decoded surface, planes in each description are adaptively compressed according to the channel error condition. Then, the two compressed bitstreams are transmitted over distinct channels to the decoder. At the decoder, if both channels are available, the two bitstreams are decoded and merged together to reconstruct a high quality surface. If only one channel is available, we employ a hole filling method to fill visual holes and reconstruct a smooth 3D surface. Therefore, the proposed algorithm provides an acceptable 3D surface, even when one channel is totally lost.

An efficient multi-hypothesis error concealment algorithm for H.264 video is proposed in this work. The proposed algorithm temporally conceals a lost block by combining several hypothesis blocks in the previous frame. We investigate the error recovery performance according to the number of hypotheses and the weighting coefficients. Simulation results demonstrate that the proposed algorithm provides better performance than the conventional error concealment algorithm, although the additional complexity requirement is negligible.

Invariant image features can be used to carry watermarks so as to improve the robustness of the watermarks against geometric transformations. However, most previous watermarking algorithms using invariant features are still sensitive to cropping attacks and combinations of rotation, scaling, and translation (RST) attacks. To improve the resilience against these attacks, we propose a multi-bit image watermarking algorithm using local Zernike moments (LZMs). The magnitude of LZMs are dither-modulated to embed watermark bits. To achieve scale invariance, we restore the original sampling rate using invariant centroid and geometric moments. Simulation results demonstrate that the proposed watermarking algorithm is robust against various geometric attacks as well as signal processing attacks.

In this work, we develop a new motion-compe (MC) interpolation algorithm to enhance the temporal resolution of video sequences. First, we propose the bilateral motion estimation scheme to obtain the motion field of an interpolated frame without yielding the hole and overlapping problems. Then, we partition a frame into several object regions by clustering motion vectors. We apply the variable-size block MC (VS-BMC) algorithm to object boundaries in order to reconstruct edge information with a higher quality. Finally, we use the adaptive overlapped block MC (OBMC), which adjusts the coefficients of overlapped windows based on the reliabilities of neighboring motion vectors. The adaptive OBMC (AOBMC) can overcome the limitations of the conventional OBMC, such as over-smoothing and poor de-blocking. Experimental results show that the proposed algorithm provides a better image quality than conventional methods both objectively and subjectively

A novel image compression algorithm based on generalized principal component analysis (GPCA) is proposed in this work. Each image block is first classified into a subspace and is represented with a linear combination of the basis vectors for the subspace. Therefore, the encoded information consists of subspace indices, basis vectors and transform coefficients. We adopt a vector quantization scheme and a predictive partial matching scheme to encode subspace indices and basis vectors, respectively. We also propose a rate-distortion optimized quantizer to encode transform coefficients efficiently. Simulation results demonstrate that the proposed algorithm provides better compression performance than JPEG, especially at low bitrates.

A progressive compression and transmission algorithm for Point-Texture 3-D images is proposed in this work. The proposed algorithm represents a Point Texture image hierarchically using an octree. The geometry information in octree nodes is encoded by the predictive partial matching (PPM) method, while the color information is encoded using the discrete cosine transform (DCT). The encoder achieves the progressive transmission of the 3-D image by transmitting the octree nodes in a top-down manner. We develop a transmission scheme, based on the rate-distortion (R-D) optimization in order to maximize the image quality subject to a given bit budget. Extensive simulation results demonstrate that the proposed algorithm is an efficient method for progressive transmission of 3-D data.

A video fingerprinting algorithm based on frame skipping is proposed in this work. At the encoder side, we skip video frames according to a pattern, specified by a fingerprint. The skipping pattern is designed such that the degradation of video quality is negligible. At the decoder side, the fingerprinted sequence is compared with the original sequence and the skipping pattern is reconstructed. Then, the watermark codeword is decoded from the skipping pattern. Since the selection of skipped frames is a combinatorial problem, the proposed algorithm can provide a large capacity for embedded information. Also, the proposed algorithm is inherently robust against spatial attacks, such as geometrical attacks and spatial filtering attacks. To improve the robustness against temporal attacks, such as frame rate reduction, we introduce the notion of error correction coding into the watermarking embedding. Experimental results show that the proposed algorithm is very robust against various attacks, while satisfying the transparency of embedded watermarks.

Predictive compression algorithms for geometry, color and normal data of three-dimensional (3-D) mesh models are proposed in this work. In order to eliminate redundancies in geometry data, we predict each vertex position by exploiting the position and angle information in neighboring triangles. To compress color data, we propose a mapping table scheme that compresses frequently recurring colors efficiently. For normal data, we propose an average predictor and a 6-4 subdivision quantizer to improve coding gain. Simulation results demonstrate that the proposed algorithm provides better performance than the MPEG-4 standard for 3-D mesh model coding (3-DMC).

A multihypothesis motion compensated prediction (MHMCP) scheme, which predicts a block from a weighted superposition of more than one reference blocks, is proposed and analyzed for error resilient visual communication in this research. By combining these reference blocks effectively, MHMCP can enhance the error resilient capability of compressed video as well as achieve a coding gain. In particular, we investigate the error propagation effect in the MHMCP coder and analyze the rate-distortion performance in terms of the hypothesis number and hypothesis coefficients. It is shown that MHMCP suppresses the short-term effect of error propagation more effectively than the intra-refreshing scheme.

Two novel error concealment techniques are proposed for video transmission over noisy channels in this work. First, we present a spatial error concealment method to compensate a lost macroblock in intra-coded frames, in which no useful temporal information is available. Based on selective directional interpolation, our method can recover both smooth and edge areas efficiently. Second, we examine a dynamic mode-weighted error concealment method for replenishing missing pixels in a lost macroblock of inter-coded frames. Our method adopts a decoder-based error tracking model and combines several concealment modes adaptively to minimize the mean square error of each pixel. The method is capable of concealing lost packets as well as reducing the error propagation effect. Extensive simulations have been performed to demonstrate the performance of the proposed methods in error-prone environments

We propose an algorithm that represents three-dimensional dynamic objects with a semi-regular mesh sequence and compresses the sequence using the spatiotemporal wavelet transform. Given an irregular mesh sequence, we construct a semi-regular mesh structure for the first frame and then map it to subsequent frames based on the hierarchical motion estimation. The regular structure of the resulting mesh sequence facilitates the application of advanced coding schemes and other signal processing techniques. To encode the mesh sequence compactly, we develop an embedded coding scheme, which supports signal-to-noise ratio and temporal scalability modes. Simulation results demonstrate that the proposed algorithm provides significantly better compression performance than the static mesh coder, which encodes each frame independently.

An error resilient three-dimensional (3-D) mesh coding system is proposed in this paper. The encoder uses a shape adaptive data partitioning scheme to alleviate the effect of error propagation. An input mesh surface is coarsely divided into smooth and detailed regions, and each region is further divided into partitions of similar sizes. Then, those partitions are progressively compressed and their joint boundaries are compressed using the boundary edge collapse rule. At the decoder, the boundary edge collapse rule facilitates the seamless assembly of the partitions. When no data is available for a partition due to transmission errors, we employ a concealment scheme based on the projection onto convex sets (POCS) theory. Simulation results demonstrate that the proposed algorithm reconstructs 3-D mesh surfaces faithfully even in severe error prone environments

A unified approach to rate-distortion (R-D) optimized compression and view-dependent transmission of three-dimensional (3-D) normal meshes is investigated in this work. A normal mesh is partitioned into several segments, which are then encoded independently. The bitstream of each segment is truncated optimally using a geometry distortion model based on the subdivision hierarchy. It is shown that the proposed compression algorithm yields a higher coding gain than the conventional algorithm. Moreover, to facilitate interactive transmission of 3-D data according to a client's viewing position, the server can allocate an adaptive bitrate to each segment based on its visibility priority. Simulation results demonstrate that the view-dependent transmission technique can reduce the bandwidth requirement considerably, while maintaining a good visual quality.

We propose a lossless compression algorithm for three-dimensional point data in graphics applications. In typical point representation, each point is treated as a sphere and its geometrical and normal data are stored in the hierarchical structure of bounding spheres. The proposed algorithm sorts child spheres according to their positions to achieve a higher coding gain for geometrical data. Also, the proposed algorithm compactly encodes normal data by exploiting high correlation between parent and child normals. Simulation results show that the proposed algorithm saves up to 60% of storage space.

The relationship between the error propagation effect and the hypothesis coefficients of the multihypothesis motion compensated prediction (MHMCP) is analyzed in this work. MHMCP enhances the error resilience of compressed video but demands a higher bit rate for the motion information. We study the rate-distortion performance of MHMCP in an error prone environment and then perform extensive experiments to conform the analytical results. Several design guidelines for MHMCP coders are drawn based on the analytical and experimental results.

Multi-hypothesis motion compensated prediction (MHMCP) predicts a block from a weighted sum of multiple reference blocks in the frame buffer. By efficiently combining these reference blocks, MHMCP can provide less prediction errors so as to reduce the coding bit rates. Although MHMCP was originally proposed to achieve high coding efficiency, it has been observed recently that MHMCP can also enhance the error resilient property of compressed video. In this work, we investigate the error propagation effect in the MHMCP coder. More specifically, we study how the multi-hypothesis number as well as hypothesis coefficients influence the strength of propagating errors. Simulation results are given to confirm our analysis. Finally, several design principles for the MHMCP coder are derived based on our analysis and simulation results.

We propose an overlapped block motion compensation method, called adaptive windowing technique, to improve the performance of variable block-size motion compensation in H.264. First, we restrict the number of neighboring blocks to be overlapped. Then we design adaptive overlapping windows, where each weight is set to be inversely proportional to the distance between the current pixel and the neighboring block. The weights can be computed effciently in both the encoder and the decoder. Also, to further improve the prediction performance, we introduce the notion of reliability of a motion vector based on the block size, and fine-tune the weights according to the reliability. Extensive simulation results show that the proposed algorithm improves the performance of H.264 both objectively and subjectively.

A lossless compression algorithm of 3D point data is proposed in this work. QSplat is one of the efficient rendering methods for 3D point data. In QSplat, each point is assigned a sphere, and the geometry and normal data are stored in the hierarchical structure of bounding spheres. To compress QSplat data, child spheres are sorted based on their limit radii to constrain the indices for the geometry data. Then, the radii and the positions of spheres are encoded separately using the reduced index sets. Also, each normal is encoded using the parent normal context, and the normal indices are reduced by the normal cone information. Simulation results show that the proposed algorithm achieves a high compression ratio by combining the reduced index sets with the context-based entropy coding.

In this paper, we develop a tree-structured predictive partial matching (PPM) scheme for progressive compression of PointTexture images. By incorporating PPM with tree-structured coding, the proposed algorithm can compress 3D depth information progressively into a single bitstream. Also, the proposed algorithm compresses color information using a differential pulse coding modulation (DPCM) coder and interweaves the compressed depth and color information effciently. Thus, the decoder can reconstruct 3D models from the coarsest resolution to the highest resolution from a single bitstream. Simulation results demonstrate that the proposed algorithm provides much better compression performance than a universal Lempel-Ziv coder, WinZip.

A view-dependent graphics streaming scheme is proposed in this work that facilitates interactive streaming and browsing of three-dimensional (3-D) graphics models. First, a 3-D model is split into several partitions. Second, each partition is simplified and coded independently. Finally, the compressed data is sent in order of relevance to the user's requests to maximize visual quality. Specifically, the server can transmit visible parts in detail, while cutting out invisible parts. Experimental results demonstrate that the proposed algorithm reduces the required transmission bandwidth, and provides an acceptable visual quality even at low bit rates.

A low complexity spatial-domain error concealment method is proposed to reconstruct still images and intra-coded (I) frames in video when they are transmitted through unreliable channels. The proposed concealment algorithm works with the following steps. First, missing edges in a lost macroblock (MB) are detected and recovered using gradient data. Then, the lost MB is implicitly divided into several segments along the recovered edges. Finally, each pixel in a segment is directionally interpolated from boundary pixels adjacent to the segment. Experimental results show that the proposed algorithm can recover high as well as low frequency information in lost MBs and provide better visual quality in comparison with the conventional spatial domain interpolation.

Three-dimensional (3D) mesh models have attribute data, such as colors, normal vectors, and texture coordinates to render or shade the surface of the mesh. Although several coding schemes have been developed to represent the topology and geometry information of the 3D mesh, coding of the attribute data has received less attention. In this paper, we propose a new predictive coding scheme for colors and normal vectors of the 3D mesh model, where we predict colors and normals based on several ancestors along the vertex ordering. In order to encode the color information, we define a mapping table that specifies how colors are mapped into other vertices. The mapping table can represent frequently occurring color patterns efficiently. For normal vectors, we also propose an average predictor and the 6-4 subdivision quantizer in the spherical coordinate system. The proposed scheme has demonstrated good coding efficiency for various VRML test data.

Compressed bit streams of 3D mesh data are highly vulnerable to transmission errors due to predictive coding and entropy coding. In this work, we propose an error resilient and progressive coding scheme for 3D meshes. To make a compressed bit stream robust to transmission errors, 3D mesh data is partitioned into several segments, which are then encoded independently. In the decoder, a novel error concealment scheme is employed to improve the visual quality of corrupted segments. Simulation results show that the proposed algorithm provides a good quality reconstruction even in severe error conditions.

In this work, we propose a multi-hypothesis error concealment algorithm, which replaces a lost block with a weighted superposition of more than two reference blocks in previous frames. Three methods are developed to find the set of reference blocks and determine the weighting coefficients. These methods are implemented based on H.26L standard, and their performances are evaluated. It is shown that the proposed multi-hypothesis algorithm provides up to 1.5 dB better performance than the conventional single-hypothesis concealment algorithm.

A low complexity spatial-domain error concealment method is proposed to reconstruct still images and intra-coded (I) frames in video when they are transmitted through unreliable channels in this work. The proposed concealment algorithm works with the following steps. First, missing edges in a lost macroblock (MB) are detected and recovered using gradient data. Then, the lost MB is implicitly divided into several segments along the recovered edges. Finally, each pixel in a segment is directionally interpolated from boundary pixels adjacent to the segment. Experimental results show that the proposed algorithm can recover high as well as low frequency information in lost MBs and provide better visual quality in comparison with the conventional spatial domain interpolation.

The signal-to-interference-plus-noise-ratio performance of the multistage linear parallel and successive interference cancellers (LPIC and LSIC) in a long-code code-division multiple-access system is analyzed using a graphical approach. The decision statistic is modeled as a Gaussian random variable, whose mean and variance can be expressed as functions of moments of R for the LPIC and L for the LSIC, respectively, where R is the correlation matrix of signature sequences and L is the strict lower triangular part of R. Since the complexity of calculating these moments increases rapidly with the growth of the stage index, a graphical representation of moments is developed to facilitate the computation. Propositions are presented to relate the moment calculation problem to several well-known problems in graph theory, i.e., the coloring, the graph decomposition, the biconnected component finding, and the Euler tour problems. It is shown that the derived analytic results match well with simulation results.

A dynamic mode-weighted error concealment method is proposed for video packets transmitted over noisy channels. We first introduce two error concealment approaches. One is to reconstruct lost pixels by interpolating candidate pixels indicated by neighboring motion vectors. The other is to estimate the motion vector by a side matching algorithm. Based on the two error concealment approaches, four corrupted block reconstruction modes are described. Then, the value of an erroneous pixel is replaced by a weighted sum of those reconstructed by two modes. The property of the weighted sum is analyzed. It is shown that the optimal weighting coefficients can be expressed as a formula in terms of the error variance and the correlation coefficients associated with the reconstruction modes. Furthermore, based on the decoder-based error tracking model, these weighting coefficients are dynamically updated to minimize the instant propagation and concealment error variance. Extensive simulations are provided to demonstrate that the proposed method can lead to a satisfying performance in an error-prone environment.

In this paper, we propose a progressive compression algorithm for three-dimensional (3D) mesh data, which can efficiently encode the connectivity and geometry information. Using cosine index predictor and vertex confinement rule, the proposed scheme encodes the connectivity information compactly. For the geometry-information coding, a 2-stage geometry predictor is proposed to reduce the prediction error more effectively than the conventional techniques. Simulation results show that the proposed algorithm yields better coding performance than the conventional algorithm.

We propose a progressive encoding algorithm for 3D voxel surfaces, based on the pattern code representation (PCR). In PCR, a voxel surface is represented by a series of pattern codes. It can achieve coding gain, since the pattern codes are highly correlated. In the multi-resolution framework, the coding gain can be further improved by the decimation constraint from the lower resolution surface. Furthermore, an alternative patternbook is adopted for voxels within smooth regions to reduce the number of possible pattern codes. Simulation results show that the proposed algorithm provides better performance than the conventional PCR algorithm.

The downlink SINR performance for multi-rate W-CDMA systems is derived in this research when the concatenated OVSF/truncated PN code is used. The derivation is complicated by the fact that the auto-correlation term of the concatenated OVSF/truncated PN code is not a random noise. It is shown that this auto-correlation term can be approximated as a binomial random variable. The analytic SINR performance can be obtained based on this approximation, and it matches well with the numerical result.

A space-time coded orthogonal frequency-division multiplexed (STC-OFDM) system is proposed to transmit layered video signals over wireless channels. The space-time coding can provide a diversity gain for the multiple-antenna system while OFDM can alleviate the frequency-selective fading effect. An input video sequence is compressed and partitioned into layers with different priorities. Then, an unequal error protection scheme based on the product Reed-Solomon (RS) codes is employed to provide different levels of protection to different layers. At the receiver, a minimum mean square error (MMSE) detector with interference cancellation (IC) is integrated with the space-time decoder to reconstruct the desired signal effectively. We derive an analytic bound for the error probability, and conduct experiments for the transmission of the H.263 video bitstream. It is shown that the proposed scheme enhances the PSNR performance significantly in the environment with a moderate signal-to-noise ratio.

We propose an efficient geometry compression algorithm for three-dimensional (3-D) mesh sequences based on the two-stage vertex-wise motion vector (MV) prediction. In general, the MV of a vertex is highly correlated to those of the adjacent vertices. To exploit this high correlation, we define the neighborhood of a vertex, and predict the MV of the vertex from those of the neighborhood. The error vectors are related to the local shape changes of 3-D objects, and still have redundancy. To remove the redundancy, the error vectors are also predicted, at the second stage, spatially or temporally by using a rate-distortion optimization technique. It is shown that the proposed algorithm has simpler structure than the existing segment-based algorithm, and yields better compression performance.

In this paper, we propose a lossless compression algorithm for three-dimensional (3-D) binary voxel surfaces, based on the pattern code representation (PCR). In PCR, a voxel surface is represented by a series of pattern codes. The pattern of a voxel v is defined as the 3 x 3 x 3 array of voxels, centered on v. Therefore, the pattern code for informs of the local shape of the voxel surface around . The proposed algorithm can achieve the coding gain, since the patterns of adjacent voxels are highly correlated to each other. The performance of the proposed algorithm is evaluated using various voxel surfaces, which are scan-converted from triangular mesh models. It is shown that the proposed algorithm requires only 0.5 approximately 1 bits per black voxel (bpbv) to store or transmit the voxel surfaces.

In this paper, we propose a new geometry compression algorithm for 3D mesh sequences, based on vertex-wise motion vector (MV) prediction. In general, the MV of a vertex is highly correlated to those of adjacent vertices. To exploit the high correlation, we define a neighborhood of a vertex, and predict the MV of the vertex from those of the neighborhood. It is shown that the proposed algorithm has simpler structure than the existing segment-based algorithm. Furthermore, simulation results demonstrate that the proposed algorithm yields better compression performance than the existing algorithm.

This paper proposes a motion compensation (MC) algorithm for robust transmission of video sequence, called the double-vector motion compensation (DMC). In the DMC, each block B in a frame is predicted from the weighted superposition of two blocks in the previous two frames, using two motion vectors. Therefore, when one of these two blocks is corrupted during the transmission, the decoder can efficiently suppress its error propagation to the subsequent frames, by predicting B only from the other block. It is shown by analysis that the DMC algorithm yields significantly lower error bounds for the subsequent frames than the conventional MC technique. Furthermore, the DMC algorithm can be combined with an effective concealment algorithm, which is capable of recovering very severe transmission errors, such as loss of an entire frame. A complete video coder, based on the DMC, is implemented by modifying the MC syntax of the H.263 standard and tested intensively in a realistic error prone environment. It is shown that the proposed algorithm provides much better objective and subjective performances than the H.263 coder in the error-prone environment.

In many video-coding standards, the motion vector field is one of the most important data in the compressed bitstream, and its loss can lead to severe degradation in the decoded picture quality. We propose the multiple description motion coding (MDMC) algorithm to enhance the robustness of the motion vector field against transmission errors. In MDMC, the motion vector field is encoded into two descriptions, which are transmitted over distinct channels to the decoder. The decoder is designed to provide an acceptable quality prediction image, even if one of the descriptions is lost during the transmission. Moreover, the decoder can reconstruct a higher quality prediction image, when both the descriptions are received without error. A complete multiple description video coder, based on the MDMC, is implemented by modifying the syntax of the H.263 standard, and tested intensively in a realistic error-prone environment. It is shown that the proposed algorithm provides much better objective and subjective performances than the H.263 coder in the error-prone environment.

Several techniques based on the multiple reference frame scheme have been proposed to improve the motion prediction gain. Though these techniques yield higher prediction gain than the single reference frame scheme, they require tremendous computational complexity during the motion search procedure. Besides, blocking artifacts may be visible along the block boundaries, since each macroblock is predicted independently of its neighbors. To overcome these drawbacks, this paper proposes a novel motion compensation algorithm, based on the double reference frame (DRF), the double motion vector (DMV), and the searching position shifting (SPS) schemes. First, to reduce the motion vector bitrate and the computational complexity of motion search procedure, we constrain the number of reference frames to 2, and use only two motion vectors per block. Second, to alleviate the blocking artifacts and to get the better pel prediction, the searching position shifting scheme is introduced. Experimental results demonstrate that the proposed algorithm yields a 3-4 dB higher prediction gain than the single reference frame scheme. The subjective quality is also improved by alleviating the blocking artifacts.

This paper proposes a novel algorithm for robust transmission of video sequences, based on the multiple description motion coding (MDMC). In the MDMC, the motion vector field is encoded into two descriptions, which are transmitted over distinct channels to the decoder. The decoder is designed to provide an acceptable quality prediction image, even if one of the descriptions is lost during the transmission. Moreover, the decoder reconstructs a higher quality prediction image, if both the descriptions are received without error. A complete multiple description video coder is implemented by modifying the syntax of the H.263 standard. Intensive simulation results show that the proposed algorithm provides much better performance than the H.263 coder in realistic error prone environment.

The motion compensation-discrete cosine transform (MC-DCT) coding is an efficient compression technique for a digital video sequence. However, the compressed video signal is vulnerable to transmission errors over noisy channels. In this paper, we propose a robust video transmission algorithm, which protects the compressed video signal by inserting redundant information at the source level. The proposed algorithm encodes every lth frame in the semi-intra frame (S-frame) mode, in which the redundant parity-check DC coefficients (PDCs) are systematically inserted into the compressed bitstream. Then, the decoder is capable of recovering very severe transmission errors, such as loss of an entire frame, in addition to detecting the errors effectively without requesting any information from external devices. The proposed algorithm is implemented based on the H.263 coder, and tested intensively in realistic error prone environment. It is shown that the proposed algorithm provides much better objective and subjective performances than the conventional H.263 coder in the error prone environment.

Motion compensation-discrete cosine transform (MC-DCT) coding is an efficient compression technique for digital video sequences. However, the compressed video signal is vulnerable to transmission errors over noisy channels. In this paper, we introduce a novel concept of parity-check motion vector (PMV) into the MC-DCT coder in order to improve its error robustness. By inserting the redundant PMVs systematically into the compressed bitstream, the proposed algorithm is capable of recovering very severe transmission errors, such as loss of an entire frame, in addition to detecting the errors effectively without requesting any information from external devices. The proposed algorithm is implemented based on the H.263 coder, and tested intensively in a realistic error prone environment. It is shown that the proposed algorithm provides much better objective and subjective performances than the conventional H.263 coder in the error prone environment.

We propose a novel algorithm for fractal video sequence coding, based on the circular prediction mapping and the noncontractive interframe mapping. The proposed algorithm can effectively exploit the temporal correlation in real image sequences, since each range block is approximated by the domain block in the adjacent frame, which is of the same size as the range block. The computer simulation results demonstrate that the proposed algorithm provides very promising performance at low bit rate, ranging from 40-250 kbyte/s.

We investigate the relation between VQ (vector quantization) and fractal image coding techniques, and propose a novel algorithm for still image coding, based on fractal vector quantization (FVQ). In FVQ, the source image is approximated coarsely by fixed basis blocks, and the codebook is self-trained from the coarsely approximated image, rather than from an outside training set or the source image itself. Therefore, FVQ is capable of eliminating the redundancy in the codebook without any side information, in addition to exploiting the self-similarity in real images effectively. The computer simulation results demonstrate that the proposed algorithm provides better peak signal-to-noise ratio (PSNR) performance than most other fractal-based coders.

This paper introduces a novel concept of PMV (parity motion vector) into the MC-DCT (motion compensation-discrete cosine transform) coder, in order to improve its robustness to transmission errors over noisy channels. By inserting the redundant PMVs systematically into the compressed bitstream, the proposed algorithm is capable of recovering very severe transmission errors, such as loss of an entire frame. The proposed algorithm is implemented based on the H.263 coder, and tested intensively in a realistic error prone environment. It is shown that the proposed algorithm provides much better objective and subjective performances than the conventional H.263 coder in the error prone environment.

We propose a novel fractal image compression technique, which does not require iteration at the decoder. The main problem relating to the conventional noniterative algorithm is that the smooth region cannot be coded efficiently, since the size of the range block is limited to be less than 8/spl times/8. We alleviate this problem by generating two codebooks from a planarly approximated image. In other words, the first codebook is generated by the smoothing operator for large and smooth range blocks and the second codebook is generated by the spatial contraction operator for small and active range blocks, respectively. The computer simulation results on the real images demonstrate that the proposed algorithm provides much better performance than most other fractal-based coders, in terms of the subjective quality as well as the objective quality (PSNR). Moreover, the proposed algorithm is very fast in decoding, since it does not require iteration at the decoder.