The rising incidence of kidney cancer underscores the need for precise and reproducible diagnostic methods. In particular, renal cell carcinoma (RCC), the most prevalent type of kidney cancer, requires accurate nuclear grading for better prognostic prediction. Recent advances in deep learning have facilitated end-to-end diagnostic methods using contextual features in histopathological images. However, most existing methods focus only on image-level features or lack an effective process for aggregating nuclei prediction results, limiting their diagnostic accuracy. In this paper, we introduce a novel framework, Nuclei feature Assisted Patch-level RCC grading (NuAP-RCC), that leverages nuclei-level features for enhanced patch-level RCC grading. Our approach employs a nuclei-level RCC grading network to extract grade-aware features, which serve as node features in a graph. These node features are aggregated using graph neural networks to capture the morphological characteristics and distributions of the nuclei. The aggregated features are then combined with global image-level features extracted by convolutional neural networks, resulting in a final feature for accurate RCC grading. In addition, we present a new dataset for patch-level RCC grading. Experimental results demonstrate the superior accuracy and generalizability of NuAP-RCC across datasets from different medical institutions, achieving a 6.15% improvement in accuracy over the second-best model on the USM-RCC dataset.

Diffusion models, which are state-of-the-art generative models, have been widely applied to image restoration tasks. However, most image restoration methods based on diffusion models require a large amount of computational memory, making it difficult to use them with high-resolution images. Although patch-based diffusion models have emerged to address this problem, these models are limited in effectively mitigating boundary artifacts and producing results close to the ground truth. In this paper, we propose Patch-based Diffusion using Constrained Refinement (PD-CR) that refines the noise estimated by patch-based diffusion models to produce a restored image while keeping the luminance of the input degraded image. Leveraging patch-based diffusion models, the proposed method can handle a high-resolution image as input with minimal memory requirements. Our experiments on various image restoration tasks, such as image denoising and raindrop removal, demonstrate that the proposed method is better than or on par with the state-of-the-art methods.

Despite rapid advancements over the past several years, the conditional generative adversarial networks (cGANs) are still far from being perfect. Although one of the major concerns of the cGANs is how to provide the conditional information to the generator, there are not only no ways considered as the optimal solution but also a lack of related research. This brief presents a novel convolution layer, called the conditional convolution (cConv) layer, which incorporates the conditional information into the generator of the generative adversarial networks (GANs). Unlike the most general framework of the cGANs using the conditional batch normalization (cBN) that transforms the normalized feature maps after convolution, the proposed method directly produces conditional features by adjusting the convolutional kernels depending on the conditions. More specifically, in each cConv layer, the weights are conditioned in a simple but effective way through filter-wise scaling and channel-wise shifting operations. In contrast to the conventional methods, the proposed method with a single generator can effectively handle condition-specific characteristics. The experimental results on CIFAR, LSUN, and ImageNet datasets show that the generator with the proposed cConv layer achieves a higher quality of conditional image generation than that with the standard convolution layer.

Although the weight-sharing property of convolution is one of the major reasons for the success of convolution neural networks, the content-agnostic operation is insufficient for several tasks requiring content-adaptive processing, including panoptic segmentation. Inspired by several recent works on content-adaptive convolutions, we introduce the GuidedPAKA, the first content-adaptive convolution method specialized for panoptic segmentation. Specifically, GuidedPAKA learns the pixel-adaptive kernel attention consisting of the channel and spatial kernel attentions. Instead of commonly used self-attention operation, we guide the channel and spatial kernel attentions using their respective supervision signals, i.e., semantic segmentation maps and local instance affinities. Consequently, these kernel attentions extract features helpful for panoptic segmentation. Experimental results show that the proposed GuidedPAKA improves the performance of panoptic segmentation when integrated into the baseline model.

Scene classification is a fundamental task in the remote sensing (RS) field, assigning semantic labels to RS images. Multispectral (MS) images play an essential role in scene classification as they contain richer spectral information than red, green, blue (RGB) images. However, MS images are not always available due to the higher cost and complexity of MS sensors compared to RGB sensors. To improve scene classification performance using only RGB images, in this letter, we propose a novel MS-to-RGB knowledge distillation (MS2RGB-KD) framework that transfers MS knowledge from a teacher model to a student model. Specifically, our MS2RGB-KD drives a student model that requires only an RGB image as input to mimic the feature representations of different modalities extracted by the teacher model. Moreover, we introduce novel loss functions that encourage the student model to preserve intramodal and intermodal relationships of the feature representations in the teacher model. Experiments on the EuroSAT dataset demonstrate the effectiveness of MS2RGB-KD compared with other KD baselines.

Many convolutional neural networks (CNNs) for single image deblurring employ a U-Net structure to estimate latent sharp images. Having long been proven to be effective in image restoration tasks, a single lane of encoder-decoder architecture overlooks the characteristic of deblurring, where a blurry image is generated from complicated blur kernels caused by tangled motions. Toward an effective network architecture for single image deblurring, we present complemental sub-solution learning with a one-encoder-two-decoder architecture. Observing that multiple decoders successfully learn to decompose encoded feature information into directional components, we further improve both the network efficiency and the deblurring performance by rotating and sharing kernels exploited in the decoders, which prevents the decoders from separating unnecessary components such as color shift. As a result, our proposed network shows superior results compared to U-Net while preserving the network parameters, and using the proposed network as the base network can improve the performance of existing state-of-the-art deblurring networks.

Multi-phase computed tomography (CT) is widely adopted for the diagnosis of kidney cancer due to the complementary information among phases. However, the complete set of multi-phase CT is often not available in practical clinical applications. In recent years, there have been some studies to generate the missing modality image from the available data. Nevertheless, the generated images are not guaranteed to be effective for the diagnosis task. In this paper, we propose a unified framework for kidney cancer diagnosis with incomplete multi-phase CT, which simultaneously recovers missing CT images and classifies cancer subtypes using the completed set of images. The advantage of our framework is that it encourages a synthesis model to explicitly learn to generate missing CT phases that are helpful for classifying cancer subtypes. We further incorporate lesion segmentation network into our framework to exploit lesion-level features for effective cancer classification in the whole CT volumes. The proposed framework is based on fully 3D convolutional neural networks to jointly optimize both synthesis and classification of 3D CT volumes. Extensive experiments on both in-house and external datasets demonstrate the effectiveness of our framework for the diagnosis with incomplete data compared with state-of-the-art baselines. In particular, cancer subtype classification using the completed CT data by our method achieves higher performance than the classification using the given incomplete data.

In adversarial learning, the discriminator often fails to guide the generator successfully since it distinguishes between real and generated images using silly or nonrobust features. To alleviate this problem, this brief presents a simple but effective way that improves the performance of the generative adversarial network (GAN) without imposing the training overhead or modifying the network architectures of existing methods. The proposed method employs a novel cascading rejection (CR) module for discriminator, which extracts multiple nonoverlapped features in an iterative manner using the vector rejection operation. Since the extracted diverse features prevent the discriminator from concentrating on nonmeaningful features, the discriminator can guide the generator effectively to produce images that are more similar to the real images. In addition, since the proposed CR module requires only a few simple vector operations, it can be readily applied to existing frameworks with marginal training overheads. Quantitative evaluations on various data sets, including CIFAR-10, CelebA, CelebA-HQ, LSUN, and tiny-ImageNet, confirm that the proposed method significantly improves the performance of GAN and conditional GAN in terms of the Frechet inception distance (FID), indicating the diversity and visual appearance of the generated images.

Coarse-to-fine strategies have been extensively used for the architecture design of single image deblurring networks. Conventional methods typically stack sub-networks with multi-scale input images and gradually improve sharpness of images from the bottom sub-network to the top sub-network, yielding inevitably high computational costs. Toward a fast and accurate deblurring network design, we revisit the coarse-to-fine strategy and present a multi-input multi-output U-net (MIMO-UNet). The MIMO-UNet has three distinct features. First, the single encoder of the MIMO-UNet takes multi-scale input images to ease the difficulty of training. Second, the single decoder of the MIMO-UNet outputs multiple deblurred images with different scales to mimic multi-cascaded U-nets using a single U-shaped network. Last, asymmetric feature fusion is introduced to merge multi-scale features in an efficient manner. Extensive experiments on the GoPro and RealBlur datasets demonstrate that the proposed network outperforms the state-of-the-art methods in terms of both accuracy and computational complexity. Source code is available for research purposes at https://github.com/chosj95/MIMO-UNet.

Among the various generative adversarial network (GAN)-based image inpainting methods, a coarse-to-fine network with a contextual attention module (CAM) has shown remarkable performance. However, due to two stacked generative networks, the coarse-to-fine network needs numerous computational resources, such as convolution operations and network parameters, which result in low speed. To address this problem, we propose a novel network architecture called parallel extended-decoder path for semantic inpainting (PEPSI) network, which aims at reducing the hardware costs and improving the inpainting performance. PEPSI consists of a single shared encoding network and parallel decoding networks called coarse and inpainting paths. The coarse path produces a preliminary inpainting result to train the encoding network for the prediction of features for the CAM. Simultaneously, the inpainting path generates higher inpainting quality using the refined features reconstructed via the CAM. In addition, we propose Diet-PEPSI that significantly reduces the network parameters while maintaining the performance. In Diet-PEPSI, to capture the global contextual information with low hardware costs, we propose novel rate-adaptive dilated convolutional layers that employ the common weights but produce dynamic features depending on the given dilation rates. Extensive experiments comparing the performance with state-of-the-art image inpainting methods demonstrate that both PEPSI and Diet-PEPSI improve the qualitative scores, i.e., the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), as well as significantly reduce hardware costs, such as computational time and the number of network parameters.

Recently, convolutional neural networks (CNNs)- based automated Gleason grading system for prostate cancer has been widely researched. However, these systems still need further improvement to achieve pathologist-level performance. To this end, this paper introduces a novel self-attentive normalization (SAN) which is the first work to employ the attention mechanism for the automated Gleason grading system. Unlike conventional normalization techniques, e.g. batch normalization and instance normalization, which learn a single affine transformation, the proposed method can learn the elementwise affine transformation to focus on more informative regions of the feature map. Since SAN requires a small number of extra learning parameters, it can be integrated into existing automated Gleason grading systems seamlessly with negligible overheads. Extensive quantitative evaluations show that, by applying SAN to various CNN architectures, the diagnostic accuracy can be significantly improved. For instance, we raise VGG-16's diagnostic accuracy from 73.99% to 79.16% on the Harvard Dataverse.

In depth map super-resolution (SR), a high-resolution color image plays an important role as guidance for preventing blurry depth boundaries. However, excessive/deficient use of the color image features often causes performance degradation such as texture-copying/edge-smoothing in flat/boundary areas. To alleviate these problems, this letter presents a simple yet effective method for enhancing the performance of the SR without requiring significant modifications to the original SR network. To this end, we present a self-selective concatenation (SSC), which is a substitute for the conventional feature concatenation. In the upsampling layers of the SR network, the SSC extracts spatial and channel attention from both color and depth features such that color features can be selectively used for depth SR. Specifically, the SSC learns to use sufficient color features for rendering sharp depth boundaries, whereas their effects are reduced in smooth regions to prevent texture-copying. The proposed SSC can be included in any existing SR networks that have the encoder-decoder structure. The experimental results show that the proposed method can further improve the performances of existing SR networks in terms of the root mean squared error and peak signal-to-noise ratio.

Lossy image compression methods with deep neural network (DNN) include a quantization process between encoder and decoder networks as an essential part to increase the compression rate. However, the quantization operation impedes the flow of gradient and often disturbs the optimal learning of the encoder, which results in distortion in the reconstructed images. To alleviate this problem, this paper presents a simple yet effective way that enhances the performance of lossy image compression without imposing training overhead or modifying the original network architectures. In the proposed method, we utilize an auxiliary branch called a shortcut which directly connects the encoder and decoder. Since the shortcut does not include the quantization process, it supports the optimal learning of the encoder by flowing the accurate gradient. Furthermore, to assist the decoder which should handle additional feature maps obtained via the shortcut, we also propose a residual refinement unit (RRU) following the quantizer. The experimental results show that the image compression network trained with the proposed method remarkably improves the performance in terms of peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and multi-scale structural similarity (MS-SSIM).

Various power-constrained contrast enhance-ment (PCCE) techniques have been applied to an organic light emitting diode (OLED) display for reducing the pow-er demands of the display while preserving the image qual-ity. In this paper, we propose a new deep learning-based PCCE scheme that constrains the power consumption of the OLED displays while enhancing the contrast of the displayed image. In the proposed method, the power con-sumption is constrained by simply reducing the brightness a certain ratio, whereas the perceived visual quality is pre-served as much as possible by enhancing the contrast of the image using a convolutional neural network (CNN). Furthermore, our CNN can learn the PCCE technique without a reference image by unsupervised learning. Ex-perimental results show that the proposed method is supe-rior to conventional ones in terms of image quality assess-ment metrics such as a visual saliency-induced index (VSI) and a measure of enhancement (EME).1.

Recently, a generative adversarial network (GAN)-based method employing the coarse-to-fine network with the contextual attention module (CAM) has shown outstanding results in image inpainting. However, this method requires numerous computational resources due to its two-stage process for feature encoding. To solve this problem, in this paper, we present a novel network structure, called PEPSI: parallel extended-decoder path for semantic inpainting. PEPSI can reduce the number of convolution operations by adopting a structure consisting of a single shared encoding network and a parallel decoding network with coarse and inpainting paths. The coarse path produces a preliminary inpainting result with which the encoding network is trained to predict features for the CAM. At the same time, the inpainting path creates a higher-quality inpainting result using refined features reconstructed by the CAM. PEPSI not only reduces the number of convolution operation almost by half as compared to the conventional coarse-to-fine networks but also exhibits superior performance to other models in terms of testing time and qualitative scores.

Interactive segmentation that extracts a specific foreground selected by the user input is widely employed in many user-interactive applications such as image editing and ground-truth labeling. In general, most interactive segmentation methods iteratively refine the previously obtained result using additional user interactions because they often produce unsatisfactory results with a single user input. A recently developed convolutional neural network (CNN)-based interactive segmentation method called deep interactive object selection has achieved high segmentation accuracy with fewer user interactions than earlier non-CNN-based approaches. However, the computational efficiency of deep interactive object selection deteriorates due to the repetitive feature extraction stage for each user interaction. Furthermore, the deep interactive object selection requires graph cut as a post-processing step to refine the boundary segments. To solve this problem, this paper presents a deep CNN-based interactive segmentation method employing an effective and simple user interaction-based attention module that does not require the repetitive feature extraction. In addition, we adopt Cartesian to polar coordinate transformation to further improve the segmentation performance. Experimental results demonstrate that the proposed interactive segmentation method is superior to the conventional ones in terms of segmentation accuracy and computational efficiency.

Gastric cancer can present itself as a gastric ulcer, which can mimic a benign gastric ulcer. In this paper, we introduce an objective and precise gastric ulcer differentiation system based on deep convolutional neural network (CNN) which can support the specialists by improving the diagnostic accuracy of the endoscopic examination of gastric ulcers. We first generated a new dataset consisting of endoscopic images of gastric ulcers and their corresponding type labels obtained by biopsy. We then design various ulcer differentiation models using classification or detection networks, and evaluate the performance of the models on the new dataset. Experimental results confirm that the classification network-based method shows performance comparable to doctors' diagnosis, and the detection network-based one, which first detects ulcer regions and then determines the type of ulcer based on the detection results, exhibits the best performance. The proposed method provides an unbiased diagnosis and it outperforms endoscopic diagnoses performed by the specialists in terms of total accuracy.

In this paper, we propose a novel method to estimate the relative camera motions of three consecutive images. Given a set of point correspondences in three views, the proposed method determines the fundamental matrix representing the geometrical relationship between the first two views by using the eight-point algorithm. Then, by minimizing the proposed cost function with the fundamental matrix, the relative camera motions over three views are precisely estimated. The experimental results show that the proposed method outperforms the conventional two-view and three-view geometry-based method in terms of the accuracy.

Quadtree-based partitioning (i.e., recursively dividing a picture into square blocks) is one of the most popular partitioning-based image coding schemes because of its computational simplicity and efficient representation of partitioning. However, the rate-distortion performance of quadtree-based partitioning reaches a limit because the dependence between child blocks of different parents is not exploited. In this paper, a new bottom-up-based block partitioning method called split-and-merge is proposed. This method splits an image into multiple square blocks and merges them into nonsquare blocks to exploit the dependence between split blocks. Moreover, a modification of the conventional intra prediction and transform is employed for nonsquare blocks. The experimental results indicate that the proposed method results in an average and maximum bit rate reduction of 3.1% and 8.3%, respectively, relative to High Efficiency Video Coding intra coding.

In this paper, we propose a new pattern model, called the structural binary pattern (SBP) model, for object tracking. For the proposed SBP model, we introduce an alternate thresholding scheme to generate a set of multiple SBPs. The SBP encodes not only the binary pattern consisting of binarized differences between the average intensities of subregions within the target region, but also the spatial configuration of the subregions. With the proposed SBP model, we define a metric for similarity between the SBP models from the target and candidate for target localization, which is based on an isotropic kernel weighted Hamming distance. To further improve the tracking performance, we employ a color-based tracking method along with the SBP-based tracking method. The experimental results show that the proposed algorithm exhibits the better performance even when the object being tracked confronts drastic illumination changes, partial occlusion, a similar colored background, or low illumination as compared with conventional tracking methods.

We propose a speed up method for the Histograms of Oriented Gradients - Local Binary Pattern (HOG-LBP) based pedestrian detector. Our method is based on the two-stage cascade structure. In the first stage evaluation, instead of extracting the features from all the region inside the detection window like in the conventional method, we extract the features from the regions which best characterize the pedestrian only. By reducing the features to be evaluated, each candidate is evaluated faster. To determine which regions are best for characterizing the pedestrian, we train the AdaBoost classifier to select the blocks whose Support Vector Machine responses of the pedestrian samples are most different from the non-pedestrians. In the second stage, we simply use the conventional HOG-LBP classifier to reevaluate the candidates which pass the first stage evaluation. Experimental results show that the detection algorithm is about three times faster than the conventional HOG-LBP SVM algorithm.

Color stereopsis is a phenomenon in the human visual system (HVS) in which a long wavelength color is perceived as being located closer than a short wavelength color. Although many psychophysical studies on color stereopsis have been carried out, its practical application is not fully investigated. In this letter, we propose a new image enhancement algorithm using color stereopsis between red and blue colors. First, the relationship of red and blue colors to the depth perception is analyzed. Then, a simple but practical image enhancement algorithm is presented based on the analysis result. The experimental results demonstrate the effectiveness of the proposed algorithm.

In this paper, we propose a new sharpness enhancement algorithm for stereo images. Although the stereo image and its applications are becoming increasingly prevalent, there has been very limited research on specialized image enhancement solutions for stereo images. Recently, a binocular just-noticeable-difference (BJND) model that describes the sensitivity of the human visual system to luminance changes in stereo images has been presented. We introduce a novel application of the BJND model for the sharpness enhancement of stereo images. To this end, an overenhancement problem in the sharpness enhancement of stereo images is newly addressed, and an efficient solution for reducing the overenhancement is proposed. The solution is found within an optimization framework with additional constraint terms to suppress the unnecessary increase in luminance values. In addition, the reliability of the BJND model is taken into account by estimating the accuracy of stereo matching. Experimental results demonstrate that the proposed algorithm can provide sharpness-enhanced stereo images without producing excessive distortion.

In this paper, we present a novel depth sensation enhancement algorithm considering the behavior of human visual system (HVS) toward stereoscopic image displays. On the basis of the recent studies on the just noticeable depth difference (JNDD), which represents a threshold that a human can perceive the depth difference between objects, we modify the depth image such that neighboring objects in the depth image can have a depth value difference of at least the JNDD. This modification is modeled via an energy minimization framework using three energy terms defined as depth data preservation, depth-order preservation, and depth difference expansion. The depth data term enforces minimal changes in the depth image with an additional weighting function that controls the direction of depth changes. The depth-order term restricts the inversion of the local and global depth orders among objects, and the JNDD term leads to an increase in the depth differences between segments. Throughout subjective quality evaluation on a stereoscopic image display, it is demonstrated that the human depth sensation is effectively improved by the proposed algorithm.

In this letter, we propose an improved histogram modification based reversible data hiding technique. In the proposed algorithm, unlike the conventional reversible techniques, a data embedding level is adaptively adjusted for each pixel with a consideration of the human visual system (HVS) characteristics. To this end, an edge and the just noticeable difference (JND) values are estimated for every pixel, and the estimated values are used to determine the embedding level. This pixel level adjustment can effectively reduce the distortion caused by data embedding. The experimental results and performance comparison with other reversible data hiding algorithms are presented to demonstrate the validity of the proposed algorithm.

More and more applications are being discovered for scalable video coding (SVC) in network-adaptive video streaming and storage systems. In the SVC encoder, an exhaustive search technique is employed to select the best coding mode for each macro block. This technique achieves the highest possible coding efficiency, but it results in extremely large encoding time. In this paper, we propose an adaptive fast mode decision algorithm for inter layer coding for spatial scalability and coarse grain quality scalability (CGS). In the proposed method, we select candidate modes adaptively for estimation in enhancement layer using the information of base layer and rate distortion (RD) cost of base layer skip mode. RD cost of base layer skip mode gives the correlation information of enhancement layer with base layer and base layer mode is used to reduce the number of candidate modes. We verify that the overall encoding time can be reduced up to 63 % through the analysis of experimental results.

In scalable video coding (SVC) as an extension of H.264/advanced video coding (AVC), a computationally expensive exhaustive mode decision is employed to select the best coding mode for each macroblock (MB). In order to reduce computational complexity, we propose a fast mode decision algorithm for SVC which uses an all-zero block (AZB) detection. Based on the empirical analysis of the inter-layer correlation of the AZB, the MB at the enhancement layer (EL) is predicted whether to be the AZB. Then, only predicted MBs are examined by the AZB detection algorithm. Since the mode decision can be terminated by detecting the AZB, we determine the processing order of the candidate modes at the EL in order to terminate the mode decision in the early stage. The proposed algorithm can be combined with other conventional fast mode decision methods in order to further reduce the computational complexity of those methods. Experimental results show that the proposed algorithm can significantly speed up the encoding process, especially for low bit-rate video sequences, without deteriorating the coding efficiency of SVC.

In this paper, a novel content-aware image resizing method based on wavelet analysis is proposed. We estimate the local energy map of an image by weighing its multiscale subbands appropriately. Based on the energy map, the image is resized by repeatedly carving out or inserting in a connected path of pixels which is least significant in terms of the energy. Since wavelet analysis is similar to the way the human visual system operates, the obtained energy map reflects human perception with fidelity, and thus, the semantic information in the image can be preserved faithfully in the resizing process. The experimental results show that the proposed method produces higher subjective quality images than scaling and conventional content-aware image resizing techniques.

In this letter, we present a novel image restoration algorithm to deblur the image without estimating the image blur. The proposed algorithm performs deblurring by merging differently blurred multiple images in the spectrum domain using the fuzzy projection onto convex sets (POCS). Experimental results demonstrate that the merged single image contains much less blur than the multiple blurred images.

In this letter, we present a hybrid motion compensation scheme integrating overlapped block motion compensation (OBMC) and control grid interpolation (CGI), called irregular grid-based OBMC (IG-OBMC). The objective of this letter is to develop both subjectively and objectively improved motion compensation scheme by combining distinct advantages of OBMC and CGI without additional motion vectors or a complicated motion estimation processing. To evaluate the performance, IG-OBMC is applied to error concealment. The simulation results demonstrate that the proposed hybrid motion compensation method achieves a better performance than the conventional motion compensation methods.

The scalable video coding (SVC) standard adopts a simple interlayer intra prediction (ILIP) method for encoding scalable video sequences. In the conventional ILIP, a prediction signal for the macroblock (MB) of the enhancement layer (EL) is obtained by simply upsampling the colocated block of the base layer (BL). We propose an improved ILIP method by generalizing the original one adopted in the SVC standard. In the proposed ILIP method, the MB of the EL is predicted using all MBs of the BL. Experimental results show that the proposed algorithm can reduce the bit rate by 1.91% to 6.44%, as compared with the conventional ILIP, while the average PSNR is not decreased.

In scalable video coding (SVC), an exhaustive mode decision is performed to search for the best mode at each macroblock. Although this method achieves an optimal trade-off between rate and distortion, it introduces an extreme computational burden on the encoder. In this letter, we propose a fast mode decision algorithm that can reduce the computational load of the mode decision for SVC. We statistically derive the expectation of the rate-distortion cost (RDcost) increase caused by skipping each mode in the mode decision. In the proposed algorithm, the encoder performs the mode decision using a small number of modes that are determined based on the expected increase of the RDcost. Experimental results show that the proposed algorithm can reduce the computational complexity significantly with negligible video quality degradation and bitrate increment.

As voice over IP (VoIP) services are becoming more widespread, UDP-restricted stateless firewalls (URSFs) have emerged as one of the main hurdles for deploying the service. By overcoming this obstacle, the new VoIP service Skype has gained rapid popularity. To traverse URSFs, Skype uses an abnormal transport method, namely voice over TCP (VoTCP). This paper investigates the effects of VoTCP on communication quality, and examines its operational network conditions. The experimental results show that the adoption of TCP for voice traffic causes a remarkable deterioration in the quality of service; however, VoTCP is applicable in a relatively clean network, such as the Skypepsilas VoTCP section between a client and the clientpsilas super node. More specifically, if the jitter buffer size is adequately tuned, VoTCP can operate at a random packet loss probability of less than 1.6%.

In this paper, an efficient bitstream switching using the progressive S-frames is presented. The progressive S-frame approach can effectively switch two pre-encoded streams with different quantization steps by reducing progressively the quantization mismatch. The progressive S-frames can provide better coding efficiency than a single S-frame. The simulation results show that the proposed method is useful for the bit rate adaptation in heterogeneous networks.

In this work, we develop a new motion-compe (MC) interpolation algorithm to enhance the temporal resolution of video sequences. First, we propose the bilateral motion estimation scheme to obtain the motion field of an interpolated frame without yielding the hole and overlapping problems. Then, we partition a frame into several object regions by clustering motion vectors. We apply the variable-size block MC (VS-BMC) algorithm to object boundaries in order to reconstruct edge information with a higher quality. Finally, we use the adaptive overlapped block MC (OBMC), which adjusts the coefficients of overlapped windows based on the reliabilities of neighboring motion vectors. The adaptive OBMC (AOBMC) can overcome the limitations of the conventional OBMC, such as over-smoothing and poor de-blocking. Experimental results show that the proposed algorithm provides a better image quality than conventional methods both objectively and subjectively

This paper presents an H.264/AVC baseline profile decoder based on a SoC platform design methodology. The overall decoding throughput is increased by optimized software and a dedicated hardware accelerator. We minimize the number of bus accesses and use macroblock (MB) level pipeline processing techniques to achieve a real time operation. We implemented and verified a prototype on a SoC platform with a 32-bit RISC CPU core and FPGA module. Our design can process up to 20 frames/sec with QCIF (176/spl times/144). The proposed architecture can be easily applied to many mobile video application areas such as a digital camera and a DMB (digital multimedia broadcasting) phone.

In this work, we propose a hybrid motion compensation integrating the advantages of control grid interpolation (CGI) and overlapped block motion compensation (OBMC). We consider control points of CGI and overlapped window of OBMC as sampling points and spread function of motion vector, respectively. Then, a whole motion field in a frame is composed of motion vectors on sampling points and their spreadings. In this view-point, the conventional OBMC is considered as regular grid based OBMC (RG-OBMC) while the proposed OBMC is called irregular grid based OBMC (IG-OBMC). Experimental results demonstrate that the proposed IG-OBMC achieves improved visual quality as well as better PSNR performance than conventional motion compensation methods.

Sample and rank selection probabilities are important in the analysis of weighted median filters as well as in comparing linear and nonlinear filters. We propose a new, efficient algorithm to calculate these probabilities for weighted median filters. The computational savings of this new algorithm are substantial and compare favorably with other known algorithms.

We propose a multilayered video coding scheme based on motion estimation which enables a decoder to dynamically change its temporal and spatial resolution during playback. In the proposed scheme, a new motion-prediction structure with a temporal hierarchy of frames is adopted to afford temporal resolution scalability and the wavelet decomposition with a new intra-update algorithm is used to offer spatial scalability. Experimental results show that the proposed scheme exhibits a higher compression ratio than conditional replenishment schemes because it further reduces the temporal redundancy using motion estimation. Also the proposed intra-update technique enables adaptation to dynamic change of spatial resolution with the effective follow-up of prediction in the decoder while preventing the large peaks in bit rate. Therefore, the proposed scheme is expected to be effectively used in heterogeneous environments such as the Internet, ATM, and wireless networks where dynamic scalability and interoperability are required.

A method for adaptation of the basis matrix of the gray-scale function processing (FP) opening and closing under the least mean square (LMS) error criterion is presented. We previously proposed the basis matrix for efficient representation of opening and closing (see IEEE Trans. Signal Processing, vol.43, p.3058-61, Dec. 1995 and IEEE Signal Processing Lett., vol.2, p.7-9, Jan. 1995). With this representation, the opening and closing operations are accomplished by a local matrix operation rather than cascade operation. Moreover, the analysis of the basis matrix shows that the basis matrix is skew symmetric, permitting to derive a simpler matrix representation for opening and closing operators. Furthermore, we propose an adaptation algorithm of the basis matrix for both opening and closing. The LMS and backpropagation algorithms are utilized for adaptation of the basis matrix. At each iteration of the adaptation process, the elements of the basis matrix are updated using the estimation of gradient to decrease the mean square error (MSE) between the desired signal and the actual filter output. Some results of optimal morphological filters applied to two-dimensional (2-D) images are presented.

A real-time implementation method for the most general morphological system, the so-called grayscale function processing (FP) system is presented. The proposed method is an extension of our previous works (1993, 1995) using the matrix representation of the FP system with a basis matrix (BM) and a block basis matrix (BBM) composed of grayscale structuring elements (GSE). In order to further improve the computational efficiency of the basis matrix representation, we propose recursive algorithms based on the observation of the BM and BBM. The efficiency of the proposed algorithms is gained by avoiding redundant steps in computing overlapping local maximum or minimum operations. It is shown that, with the proposed scheme, both opening and closing can be determined in real time by 2N-2 additions and 2N-2 comparisons, and OC and CO by 4N-4 additions and 4N-4 comparisons, when the size of the GSE is equal to N. It is also shown that the proposed recursive opening and closing require only 3N-3 memory elements.

We propose a method for the real-time implementation of function processing (FP), open-closing (OC), and close-opening (CO) morphological operations. The proposed method is based on the block basis matrix (BBM), which is an extension of the basis matrix of Ko and Shridar (1990). A procedure to obtain the BBM is proposed. This matrix is skew symmetric, and some related properties lead to simplify the FP morphological operations. It is shown that the real-time calculation of open-closing and close-opening is accomplished by local matrix operations rather than cascade operations (opening/closing followed by closing/opening), thus eliminating delays and requiring less memory storage.>

The aim of this paper is to find a relationship between alternating sequential filters (ASF) and the morphological sampling theorem (MST) developed by Haralick et al. (1987). The motivation behind this approach is to take advantage of the computational efficiency offered by the MST to implement morphological operations. First, we show alternative proofs for opening and closing in the sampled and unsampled domain using the basis functions. These proofs are important because they show that it possible to obtain any level of a morphological pyramid in one step rather than the traditional two-step procedure. This decomposition is then used to show the relationship of the open-closing in the sampled and unsampled domain. An upper and a lower bound, for the above relationships, are presented. Under certain circumstances, an equivalence is shown for open-closing between the sampled and the unsampled domain. An extension to more complicated algorithms using a union of openings and an intersection of closings is also proposed. Using the Hausdorff metric, it is shown that a morphologically reconstructed image cannot have a better accuracy than twice the radius of the reconstruction structuring element. Binary and gray scale examples are presented.

The Winsorizing smoother (W smoother), which is a center weighted median (CWM) filter giving more weight only to the central value of each window, is studied. This filter can preserve image details while suppressing additive white and/or impulsive-type noise. The statistical properties of the W smoother are analyzed. It is shown that the W smoother can outperform the median filter, while its implementation is almost as simple as median filtering. Some relationships between W smoothers and other median-type filters, such as the weighted median filter and the multi-stage median filter, are derived.>

Median filtering can be viewed as operation that selects a sample from each window close to the last output. This observation results in a new edge preserving smoother called the last output reference (LOR) filter. The LOR filter is similar in function to the median or recursive median filters, but has additional advantages particularly in suppressing impulses. It has been shown that repeated applications of LOR filtering produces a sequence that is invariant to subsequent passes through the same filter. Also, it has been proven that any sequence can be converted to a locally monotone sequence by using a combination of "forward" and"backward" LOR filters. When the LOR filter is applied to an actual noisy image, it is shown to perform well.