Defocus blur, common in shallow depth-of-field photography, varies across image regions and is challenging to accurately estimate and restore. Existing deblurring methods often struggle to capture fine structural textures and do not effectively adapt to regional differences in blur. We propose Multi-Level Blur-Aware Stable Diffusion (MBSD), a novel framework that explicitly integrates regional blur recognition into a diffusion-based image restoration process. MBSD assigns blur-level labels to image patches using a Patch Blur Annotator (PBA), guiding a Multi-Scale Blur Estimator (MSBE) to predict soft blur probabilities and generate routing weights. These weights control a Blur-Adaptive Expert Mixer (BAEM), which adaptively combines features based on local blur severity. The features are then passed to a text-to-image diffusion model via a cross-attention mechanism, enabling region-specific restoration. Extensive experiments on public benchmarks demonstrate that MBSD delivers superior perceptual quality while maintaining competitive PSNR and SSIM, consistently outperforming state-of-the-art methods.

To address the challenges of stereo matching under sparse view conditions, which yield limited overlap, occlusion, and large parallax, a novel Depth Enhanced 3D Gaussian Splatting (DE-3DGS) framework is presented in this paper. By generating an intermediate virtual viewpoint image, disparity maps are obtained via trinocular stereo matching to improve depth reliability in wide baseline scenarios. The source images, together with the estimated depth map, are fused to predict the attributes and feature embeddings of the 3D Gaussian. In addition, a feature refinement module is proposed to improve the structural completeness and appearance coherence by refining incomplete regions. Experiments on THuman2.0 and THumanSit validate that the trinocular matching strategy of DE-3DGS, guided by intermediate viewpoints, significantly improves rendering quality and structural completeness in wide-baseline scenarios.

Gaze-based object manipulation intention inference is pivotal to natural and intuitive human-robot interaction. Existing methods confine spatial regularity to independent object selection and treat temporal regularity only as sequential order, thus spatio-temporal gaze regularities remain insufficiently exploited and lack a unified treatment. The objective of this study is to statistically analyze, model, and integrate gaze regularities within a unified probabilistic framework for inference. Accordingly, we propose a spatio-temporal gaze regularities guided dynamic fuzzy Bayesian network (DFBN) for intention inference. We statistically analyze the spatial gaze regularity as mutual exclusion and co-occurrence in joint object selection patterns, and the temporal regularity as duration-dependent attention with sequential dependencies. The regularities are modeled into probabilistic form, with spatio regularity modeled by autoregressive logistic regression and temporal regularity modeled by a Fuzzy Gaze-LSTM that fuses gaze duration with sequential order. These probabilistic models are fused into a likelihood modifier to generate interpretable posterior probabilities of intention, integrating a Bayesian network and sequential inference. Cross-dataset evaluations indicate stable and high performance. DFBN attains 96.47 1.75% accuracy and 96.41 1.85% F1 score, maintains accuracy on error sequences, and generalizes across younger and older groups, supporting robust intention inference. This study has the potential to inform other human-robot interaction assistance strategies by serving as intuitive gaze regularity cues.

This paper presents a novel image enhancement approach to avoid common artifacts such as over-exposure, color cast, and unnatural results. The key innovation lies in estimating the illumination map of an underexposed image using a sparse bright channel. Our approach includes an algorithm that enforces the sparsity of the inverted bright channel, enabling the indirect estimation of a coarse but suitable initial illumination map. This initial map is refined using an updated weight-constrained regularization with joint local exposure and detail feedback constraints, producing a piece-wise smooth, structure-preserving illumination map. Computer simulations show that the proposed method is competitive with or even outperforms several state-of-the-art enhancement methods in terms of both subjective and objective evaluations.

Graph regularized nonnegative matrix factorization (GNMF) has been widely used in data representation due to its excellent dimensionality reduction. When it comes to clustering polluted data, GNMF inevitably learns inaccurate representations, leading to models that are unusually sensitive to outliers in the data. For example, in a face dataset, obscured by items such as a mask or glasses, there is a high probability that the graph regularization term incorrectly describes the association relationship for that sample, resulting in an incorrect elicitation in the matrix factorization process. In this article, a novel self-initiated unsupervised subspace learning method named robust nonnegative matrix factorization with self-initiated multigraph contrastive fusion (RNMF-SMGF) is proposed. RNMF-SMGF is capable of creating samples with different angles and learning different graph structures based on these different angles in a self-initiated method without changing the original data. In the process of subspace learning guided by graph regularization, these different graph structures are fused into a more accurate graph structure, along with entropy regularization, $L_{2,1/2}$ -norm constraints to facilitate the robust learning of the proposed model and the formation of different clusters in the low-dimensional space. To demonstrate the effectiveness of the proposed model in robust clustering, we have conducted extensive experiments on several benchmark datasets and demonstrated the effectiveness of the proposed method. The source code is available at: https://github.com/LstinWh/RNMF-SMGF/.

A fundamental task in the realms of computer vision, Low-Rank Matrix Recovery (LRMR) focuses on the inherent low-rank structure precise recovery from incomplete data and/or corrupted measurements given that the rank is a known prior or accurately estimated. However, it remains challenging for existing rank estimation methods to accurately estimate the rank of an ill-conditioned matrix. Also, existing LRMR optimization methods are heavily dependent on the chosen parameters, and are therefore difficult to adapt to different situations. Addressing these issues, A novel LEarning-based low-rank matrix recovery with Rank Estimation (LERE) is proposed. More specifically, considering the characteristics of the Gerschgorin disk's center and radius, a new heuristic decision rule in the Gerschgorin Disk Theorem is significantly enhanced and the low-rank boundary can be exactly located, which leads to a marked improvement in the accuracy of rank estimation. According to the estimated rank, we select row and column sub-matrices from the observation matrix by uniformly random sampling. A 17-iteration feedforward-recurrent-mixed neural network is then adapted to learn the parameters in the sub-matrix recovery processing. Finally, by the correlation of the row sub-matrix and column sub-matrix, LERE successfully recovers the underlying low-rank matrix. Overall, LERE is more efficient and robust than existing LRMR methods. Experimental results demonstrate that LERE surpasses state-of-the-art (SOTA) methods. The code for this work is accessible at https://github.com/zhengqinxu/LERE.

Self-calibrating gaze estimation eliminates burdensome personal calibration by utilizing gaze behaviors to calibrate gaze estimation models automatically. However, they rely on viewing multiple specific scenes or sacrifice accuracy. Due to the pervasiveness of reading, we propose a pervasive and accurate self-calibrating approach that requires reading only a few lines of text naturally. This approach reformulates the calibration model as nonlinearly matching spatio-temporal reading patterns and the corresponding eye-feature patterns. The eye features are filtered into fixations by filtering nonreading data and saccades to ensure visual intake. Fixations are segmented into multiple lines by recovering the temporal reading pattern, thus simplifying the nonlinear matching into the line-to-line matching between segmented fixations and text lines, which is achieved by recovering the spatial reading pattern. Experimental results show that the proposed approach has comparable accuracy to state-of-the-art head-mounted gaze estimation methods, which require explicit calibration or multiple salient scenes.

Gaze estimation suffers from burdensome personal calibration or complex all-device calibration. Self-calibrating methods can meet this challenge but depend on scenes and sacrifice accuracy. We propose a flexible and accurate gaze estimation approach calibrated implicitly with potential gaze patterns. By constructing an optical axis projection (OAP) plane and a visual axis projection (VAP) plane simultaneously, the optical axis and the visual axis can be represented as 2-D points, i.e., the OAP and VAP, which have a similarity transformation, indicating the linear consistency of OAP patterns with gaze patterns. Hence, a 3-D gaze estimation model using the OAP as an eye feature to predict the VAP is built. The unknown parameters are calculated separately by linearly aligning OAP patterns to natural and easily detectable gaze patterns. Experimental results show that the proposed gaze estimation approach is more accurate than state-of-the-art head-mounted gaze estimation methods, which require explicit calibration or multiscene saliency.

Robust principal component analysis (RPCA) is widely studied in computer vision. Recently an adaptive rank estimate based RPCA has achieved top performance in low-level vision tasks without the prior rank, but both the rank estimate and RPCA optimization algorithm involve singular value decomposition, which requires extremely huge computational resource for large-scale matrices. To address these issues, an efficient RPCA (eRPCA) algorithm is proposed based on block Krylov iteration and CUR decomposition in this paper. Specifically, the Krylov iteration method is employed to approximate the eigenvalue decomposition in the rank estimation, which requires$O(ndrq+n(rq)^{2})$for an$(n\times d)$input matrix, in which$q$is a parameter with a small value,$r$is the target rank. Based on the estimated rank, CUR decomposition is adopted to replace SVD in updating low-rank matrix component, whose complexity reduces from$O(rnd)$to$O(r^{2}n)$per iteration. Experimental results verify the efficiency and effectiveness of the proposed eRPCA over the state-of-the-art methods in various low-level vision applications.

Brightness order reversal usually appears when two large-exposure-ratio images of a high dynamic range scene are directly fused together by an existing multi-scale exposure fusion algorithm. To address the problem, a novel neural augmented framework is introduced to interpolate an image with the medium exposure by integrating physics-driven and data-driven approaches. The physics-driven method infers high-frequency information while the data-driven approach learns remaining information for the interpolated image. The interpolated image and two large-exposure-ratio images are fused together. Experimental results show that the proposed framework can indeed solve the brightness order reversal problem for the fusion of of two large-exposure-ratio images.

Due to saturated regions of low dynamic range (LDR) images and large intensity changes among them, it is challenging to produce an information-enriched panoramic LDR image without visual artifacts from multiple geometrically synchronized LDR images with different exposures and piecewise overlapping fields of views for a high dynamic range (HDR) scene. Fortunately, the stitching of such images is innately a perfect scenario for the fusion of physics-driven and data-driven methods. Based on the insight, a novel neural augmented HDR panoramic stitching algorithm is proposed in this paper. Differently exposed panoramic LDR images are initialized by using a physics-driven method on top of the piecewise overlapping fields of views. They are then refined by a data-driven one, and finally merged together via a multi-scale exposure fusion algorithm to produce the desired panoramic LDR image. Experimental results validate the proposed algorithm11The source code and trained model will be publicly available upon the acceptance..

In recent years, remarkable results have been achieved in self-supervised action recognition using skeleton sequences with contrastive learning. It has been observed that the semantic distinction of human action features is often represented by local body parts, such as legs or hands, which are advantageous for skeleton-based action recognition. This paper proposes an attention-based contrastive learning framework for skeleton representation learning, called SkeAttnCLR, which integrates local similarity and global features for skeleton-based action representations. To achieve this, a multi-head attention mask module is employed to learn the soft attention mask features from the skeletons, suppressing non-salient local features while accentuating local salient features, thereby bringing similar local features closer in the feature space. Additionally, ample contrastive pairs are generated by expanding contrastive pairs based on salient and non-salient features with global features, which guide the network to learn the semantic representations of the entire skeleton. Therefore, with the attention mask mechanism, SkeAttnCLR learns local features under different data augmentation views. The experiment results demonstrate that the inclusion of local feature similarity significantly enhances skeleton-based action representation. Our proposed SkeAttnCLR outperforms state-of-the-art methods on NTURGB+D, NTU120-RGB+D, and PKU-MMD datasets. The code and settings are available at this repository: https://github.com/GitHubOfHyl97/SkeAttnCLR.

Model-based single image dehazing algorithms restore images with sharp edges and rich details at the expense of low PSNR values. Data-driven ones restore images with high PSNR values but with low contrast, and even some remaining haze. In this paper, a novel single image dehazing algorithm is introduced by integrating model-based and data-driven approaches. Both transmission map and atmospheric light are initialized by the model-based methods, and refined by deep learning based approaches which form a neural augmentation. Haze-free images are restored by using the transmission map and atmospheric light. Experimental results indicate that the proposed algorithm can remove haze well from real-world and synthetic hazy images.

In this letter, a new texture filter with scale-aware gradients and structural preservation is proposed. The proposed filter uses a hybrid$L_{0}$-$H^{-1} $variational model via measuring sparsity with scale-aware gradients by the$L_{0}$norm and structural fidelity by the$H^{-1}$norm with the embedded Laplacian operator. Extensive qualitative and quantitative experimental results demonstrate that the proposed method 1) smooths small-scale low/high contrast textures and intensive noise while preserving sharp and prominent structures simultaneously; 2) significantly outperforms state-of-the-art texture filtering methods; and 3) has fast convergence.

Accurate gaze estimation that frees from glints and the slippage problem is challenging. Pupil-contour-based gaze estimation methods can meet this challenge, except that the gaze accuracy is low due to neglecting the pupil’s corneal refraction This article proposes a refraction-aware gaze estimation approach using the real pupil axis, which is calculated from the virtual pupil image based on the derived function between the real pupil and the refracted virtual pupil. We present a 2-D gaze estimation method that regresses the real pupil normal’s spherical coordinates to the gaze point. The noise and outliers of calibration data are removed by aggregation filtering and random sample consensus, respectively. Moreover, we propose a 3-D gaze estimation method that transforms the real pupil axis to the gaze direction. Experimental results show that the proposed gaze estimation approach has comparable accuracy to state-of-the-art pupil-center-based gaze estimation methods, which suffer from the slippage problem.

Model-based single image dehazing algorithms restore haze-free images with sharp edges and rich details for real-world hazy images at the expense of low PSNR and SSIM values for synthetic hazy images. Data-driven ones restore haze-free images with high PSNR and SSIM values for synthetic hazy images but with low contrast, and even some remaining haze for real-world hazy images. In this paper, a novel single image dehazing algorithm is introduced by combining model-based and data-driven approaches. Both transmission map and atmospheric light are first estimated by the model-based methods, and then refined by dual-scale generative adversarial networks (GANs) based approaches. The resultant algorithm forms a neural augmentation which converges very fast while the corresponding data-driven approach might not converge. Haze-free images are restored by using the estimated transmission map and atmospheric light as well as the Koschmieder's law. Experimental results indicate that the proposed algorithm can remove haze well from real-world and synthetic hazy images.

Robust principal component analysis (RPCA) and its variants have gained vide applications in computer vision. However, these methods either involve manual adjustment of some parameters, or require the rank of a low-rank matrix to be known a prior. In this paper, an adaptive rank estimate based RPCA (ARE-RPCA) is proposed, which adaptively assigns weights on different singular values via rank estimation. More specifically, we study the characteristics of the low-rank matrix, and develop an improved Gerschgorin disk theorem to estimate the rank of the low-rank matrix accurately. Furthermore in view of the issue occurred in the Gerschgorin disk theorem that adjustment factor need to be manually pre-defined, an adaptive setting method, which greatly facilitates the practical implementation of the rank estimation, is presented. Then, the weights of singular values in the nuclear norm are updated adaptively based on iteratively estimated rank, and the resultant low-rank matrix is close to the target. Experimental results show that the proposed ARE-RPCA outperforms the state-of-the-art methods in various complex scenarios.

It is challenging to remove rain steaks from a single rainy image because the rain steaks are spatially varying in the rainy image. On top of a new insight in single image de-raining, a nonlocal de-raining algorithm is proposed in this paper to remove the rain streaks from the rainy image. The rainy image is not decomposed into different layers by the proposed algorithm. Experimental results validate the proposed algorithm.

Image texture

A small ISO and a small exposure time are usually used to capture an image in back- or low-light condition which results in an image with negligible motion blur and small noise but looks dark. In this paper, a single image brightening algorithm is introduced to brighten such an image. The proposed algorithm includes a unique hybrid learning framework to generate two virtual images with large exposure times. The virtual images are first generated via intensity mapping functions (IMFs) which are computed using camera response functions (CRFs) and this is a model-driven approach. Both the virtual images are then enhanced by using a data-driven approach, i.e. a residual convolutional neural network to approach the ground truth images. The model-driven approach and the data-driven one compensate each other in the proposed hybrid learning framework. The final brightened image is obtained by fusing the original image and two virtual images via a multi-scale exposure fusion algorithm with properly defined weights. Experimental results show that the proposed brightening algorithm outperforms existing algorithms in terms of MEF-SSIM metric.

Deep learning based methods have become dominant solutions to many image processing problems. A natural question would be "Is there any space for conventional methods on these problems?" In this paper, exposure interpolation is taken as an example to answer this question and the answer is "Yes". A new hybrid learning framework is introduced to interpolate a medium exposure image for two large-exposure-ratio images from an emerging high dynamic range (HDR) video capturing device. The framework is set up by fusing conventional and deep learning methods. Experimental results indicate that the deep learning method can be used to improve the quality of interpolated image via the conventional method significantly. The conventional method can be adopted to increase the convergence speed of the deep learning method and to reduce the number of samples which is required by the deep learning method. They compensate each other.

It is challenging to convert a hazy color image into a gray-scale image because the color contrast field of a hazy image is distorted. In this paper, a novel decolorization algorithm is proposed to transfer a hazy image into a distortionrecovered gray-scale image. To recover the color contrast field, the relationship between the restored color contrast and its distorted input is presented in CIELab color space. Based on this restoration, a nonlinear optimization problem is formulated to construct the resultant gray-scale image. A new differentiable approximation solution is introduced to solve this problem with an extension of the Huber loss function. Experimental results show that the proposed algorithm effectively preserves the global luminance consistency while represents the original color contrast in gray-scales, which is very close to the corresponding ground truth gray-scale one.

Existing multiscale exposure fusion (MEF) algorithms cannot preserve relative brightness in an image fused from two large-exposure-ratio images if high-light regions in the dark image are darker than shadow regions in the bright image. In this letter, a strategy by synthesizing a virtual image with a medium exposure is presented to brighten the high-light regions in the dark image and to darken the darkest regions in the bright image. The virtual image is generated via intensity mapping functions. In order to avoid possible color distortion in the virtual image due to one-to-many mapping, two intermediate virtual images with the same exposure time are generated by the two input images, and then merged together to produce the desired virtual image using properly defined weights. The final image is obtained by fusing the original two input images and the virtual image via a state-of-the-art MEF algorithm. Experimental results show that the relative brightness is preserved much better and the MEF-SSIM is significantly improved by the proposed algorithm.

Decolorization is to convert a color image into a gray scale image while preserve image features like salient structure and chrominance contrast. The sign of the color contrast is crucial for the decolorization algorithm and is usually determined in existing works by giving a strict defined color order or twomode weak order. In this paper, a fast computation on color order is achieved via a simple global mapping which is introduced in a linear parametric model using an extended structure transfer filter. The values of the parameters are obtained via an elegant approximation method. A local decolorization algorithm is finally designed on basis of the global linear mapping so that both color and spatial information are preserved robustly and accurately. Experimental results show that the proposed decolorization algorithms obtain a good performance among existing quality metrics for the decolorization. In addition, the proposed global decolorization algorithm is friendly to mobile devices with limited computational resource.

One key issue in content‐based image retrieval is to extract effective features so as to represent the visual content of an image. In this study, a new non‐negative sparse feature learning approach to produce a holistic image representation based on low‐level local features is presented. Specifically, a modified spectral clustering method is introduced to learn a non‐negative visual dictionary from local features of training images. A non‐negative sparse feature encoding method termed non‐negative locality‐constrained linear coding (NNLLC) is proposed to improve the popular locality‐constrained linear coding method so as to obtain more meaningful and interpretable sparse codes for feature representation. Moreover, a new feature pooling strategy named kMaxSum pooling is proposed to alleviate the information loss of the sum pooling or max pooling strategy, which produces a more effective holistic image representation and can be viewed as a generalisation of the sum and max pooling strategies. The retrieval results carried out on two public image databases demonstrate the effectiveness of the proposed approach.

In this paper, a novel extreme learning machine based online multi-label classifier for real-time data streams is proposed. Multi-label classification is one of the actively researched machine learning paradigm that has gained much attention in the recent years due to its rapidly increasing real world applications. In contrast to traditional binary and multi-class classification, multi-label classification involves association of each of the input samples with a set of target labels simultaneously. There are no real-time online neural network based multi-label classifier available in the literature. In this paper, we exploit the inherent nature of high speed exhibited by the extreme learning machines to develop a novel online real-time classifier for multi-label data streams. The developed classifier is experimented with datasets from different application domains for consistency, performance and speed. The experimental studies show that the proposed method outperforms the existing state-of-the-art techniques in terms of speed and accuracy and can classify multi-label data streams in real-time.

In this paper, we propose a noise reduced tone mapping method based on information content weights, where the perceptually unimportant pixels are smoothed during the decomposition in two steps. First, a saliency-based information content weight is introduced to give high fidelity to the data term based on the ratio of the local pixel power and the overall noise power in the base layer decomposition. Then, the detail layer is subtracted using the mutual information-based information content weight from the original image luminance and the clean base layer. Experiments show the effectiveness of the proposed method in the improvements of both signal-to-noise ratio and visual quality.

We propose a novel robust method to accurately align large-photometric-variation and noisy images emerged from high dynamic range (HDR) imaging. First, the keypoints are detected from optimal multi-binary images to eliminate the noise and photometric effects. The feature descriptor, which is translation-, rotation- and scale-invariant and robust to noise and photometric changes, is then developed, and feature matching is implemented efficiently by use of the structure information of the keypoints. Finally, mutual information (MI) is employed in the RANSAC method for homography estimation and performance assessment, which makes the results accurate and stable. Experiments carried out on synthesized images demonstrate that the proposed method is much more robust to both photometric changes and noise than the SIFT method, the state-of-the-art alignment method.

It is very challenging to accurately detect smoke from images because of large variances of smoke colour, textures, shapes and occlusions. To improve performance, the authors combine dual threshold AdaBoost with staircase searching technique to propose and implement an image smoke detection method. First, extended Haar‐like features and statistical features are efficiently extracted from integral images from both intensity and saturation components of RGB images. Then, a dual threshold AdaBoost algorithm with a staircase searching technique is proposed to classify the features of smoke for smoke detection. The staircase searching technique aims at keeping consistency of training and classifying as far as possible. Finally, dynamic analysis is proposed to further validate the existence of smoke. Experimental results demonstrate that the proposed system has a good robustness in terms of early smoke detection and low false alarm rate, and it can detect smoke from videos with size of 320 × 240 in real time.

Many recent applications require text segmentation for born-digital compound images. To this end, we propose a coarse-to-fine framework for segmenting texts of arbitrary scales and orientations in born-digital compound images. In the coarse stage, the local image activity measure is designed based upon the variation distribution of characters, to highlight the difference between textual and pictorial regions. This stage outputs a coarse textual layer including textual regions as well as a few pictorial regions with high activity. In the fine stage, a textual connected component (TCC) based refinement is proposed to eliminate the survived pictorial regions. In particular, a scale and orientation invariant grouping algorithm is proposed to adaptively generate TCCs with uniform statistical features. The minimum average distance and morphological operations are employed to assist the formation of candidate TCCs. Then, three string-level features (i.e., shapeness, color similarity, and mean activity level) are designed to distinguish the true TCCs from the false positive ones that are formed by connecting the high activity pictorial components. Extensive experiments show that the proposed framework can segment textual regions precisely from born-digital compound images, while preserving the integrity of texts with varied scales and orientations, and avoiding over-connection of textual regions.

It is known that local filtering-based edge preserving smoothing techniques suffer from halo artifacts. In this paper, a weighted guided image filter (WGIF) is introduced by incorporating an edge-aware weighting into an existing guided image filter (GIF) to address the problem. The WGIF inherits advantages of both global and local smoothing filters in the sense that: 1) the complexity of the WGIF is O(N) for an image with N pixels, which is same as the GIF and 2) the WGIF can avoid halo artifacts like the existing global smoothing filters. The WGIF is applied for single image detail enhancement, single image haze removal, and fusion of differently exposed images. Experimental results show that the resultant algorithms produce images with better visual quality and at the same time halo artifacts can be reduced/avoided from appearing in the final images with negligible increment on running times.

This letter presents a novel exposure-robust method to align differently exposed images. First, a directional mapping approach is introduced to normalize differently exposed images so as to alleviate the effect of saturation. Then, a non-parametric local binary pattern (LBP) is employed to represent intensity-invariant features of these images. An efficient two-stage alignment is proposed for motion estimation. Experiments on a variety of synthesized and real image sequences demonstrate that the proposed method is less sensitive to the reference image, and robust to 12 exposure values (EV) increments, which is superior to existing methods.

In this paper, we introduce an exposure fusion scheme for differently exposed images with moving objects. The proposed scheme comprises a ghost removal algorithm in a low dynamic range domain and a selectively detail-enhanced exposure fusion algorithm. The proposed ghost removal algorithm includes a bidirectional normalization-based method for the detection of nonconsistent pixels and a two-round hybrid method for the correction of nonconsistent pixels. Our detail-enhanced exposure fusion algorithm includes a content adaptive bilateral filter, which extracts fine details from all the corrected images simultaneously in gradient domain. The final image is synthesized by selectively adding the extracted fine details to an intermediate image that is generated by fusing all the corrected images via an existing multiscale algorithm. The proposed exposure fusion algorithm allows fine details to be exaggerated while existing exposure fusion algorithms do not provide such an option. The proposed scheme usually outperforms existing exposure fusion schemes when there are moving objects in real scenes. In addition, the proposed ghost removal algorithm is simpler than existing ghost removal algorithms and is suitable for mobile devices with limited computational resource.

In this work, we propose a method to detect visually salient objects in computer synthesized images from 3D meshes. Different from existing detection methods on graphic saliency which compute saliency based on pixel-level contrast, the proposed method computes saliency by measuring object-level contrast of each object to the other objects in a rendered image. Given a synthesized image, the proposed method first extracts dominant colors from each object, and represents each object with the dominant color descriptor (DCD). Saliency is measured as the contrast between the DCD of the object and the DCDs of its surrounding objects. We evaluate the proposed method on a data set of computer rendered images, and the results show that the proposed method obtains much better performance compared with existing related methods.

It is very challenging to synthesize a high dynamic range (HDR) image from multiple differently exposed low dynamic range images when there are moving objects in the images. This is due to the fact that the moving objects will cause ghosting artifacts to appear in the synthesized HDR image. To prevent such artifacts, a patching algorithm is required to correct motion regions such that all the moving objects are synchronized in the differently exposed images. In this paper, a new optimization problem is formulated to correct the motion regions of the multiple differently exposed images by considering both spatial and temporal consistencies. The resultant scheme is a hybrid patching scheme composed of a correction method which is an intensity mapping function at pixel level, and a hole-filling method that uses block-level template matching. The proposed patching scheme is not only robust to large intensity changes in these input images, but also at regions that are over- or underexposed. Experimental results show that the proposed method is able to prevent ghosting artifacts from appearing in the final synthesized HDR image.

In this paper, a bilateral filter in gradient domain is first proposed. It is then applied to study detail enhancement via multi-light images and noise reduction of differently exposed low dynamic range images. These two applications show that the proposed filter can be applied to extract fine details from a set of images simultaneously and to provide flexibility for noise reduction from selected areas of an image.

In a typical image synthesis where multiple differently exposed images are captured for processing, it is important to design an anti-ghost algorithm so as to prevent ghosting artifacts from appearing in the final image. An anti-ghost algorithm is usually composed of a detection module and a correction module. In this paper, a new detection module is proposed to detect non-consistent pixels of all input images without predefining any initial reference image. The proposed module is suitable when an interactive mode is desired. In addition, a bidirectional approach is introduced to correct the non-consistent pixels in the correction module. Compared with existing unidirectional correction methods, the proposed bidirectional correction approach uses information from two adjacent images of a detected image to correct its non-consistent pixels. This leads to a quality improvement in the final image.

A robust scheme is proposed to generate an anti-ghosting high dynamic range (HDR) image from a set of low dynamic range (LDR) images with different exposure times. Three major contributions of this paper are 1) a bi-directional prediction method; 2) an adaptive threshold for the classification of pixels; 3) Bayes estimator based methods for the on-line updating of predicted values and the synthesis of pixels to fill in the regions of moving objects to preserve their dynamic ranges. The proposed scheme is suitable for both static and dynamic scenes.

In this paper, we propose an intensity mapping function (IMF) based scheme to detect moving objects in a set of low dynamic range (LDR) images with different known exposure times. The objective is to remove ghosting artifacts from the eventual high dynamic range (HDR) image. Our contributions include a bidirectional similarity detection method, an adaptive threshold for movement detection, and an IMF based method for the synthesis of pixels to fill in the regions of moving objects. Experimental results show that the proposed scheme outperforms existing schemes.

This paper presents a robust and fast algorithm for automatically generating high dynamic range (HDR) images in presence of camera movement and moving objects. This scheme comprises five modules: 1) image alignment, 2) estimation of camera response function (CRF) in dynamic scenes, 3) moving object detection, 4) progressive image correction, and 5) construction of HDR images. The key advantage of the algorithm is the ability to generate HDR images without ghost artifact. The proposed algorithm is fast as it is a one-shot solution without iterative computation and post-processing or even manual operation. Experimental results demonstrate that the proposed method outperforms the existing commercial products.

A video quality monitoring system for in-service monitoring of videos is described in this paper. The technologies developed include a no-reference method for blockiness measurement that exploits the characteristics of human visual system using the concept of just-noticeable-difference. This approach works even for scenario where the blockiness is not at known fixed locations. To provide a complete over-view of the integrated video quality monitoring system, methodologies for measuring picture freeze and picture loss have also been provided. The experimental results show that the proposed video quality monitoring system gives good accuracy for artefacts detections.

In this paper, a infrared face recognition method using radiant energy conversion and curvelet transformation is proposed. Firstly, to get the stable feature of thermal face, thermal images are converted into radiant energy images according to Stefan-Boltzmann's law. Secondly, curvelet transform has better directional and edge representation abilities than widely used wavelet transformation and other classic transformations. Inspired by these attractive attributes of curvelets in sparse representation of the images, we introduce the idea of decomposing images into their curvelet subbands to extract the principal representative feature, which saves the computational complexity and storage units. Finally, the nearest neighbor classifier is chosen to get the system recognition result. The experiments illustrate that compared with traditional PCA based systems, the proposed system has better performance and requires fewer computations and memory units.

This paper describes a video quality analysis system for inservice monitoring of streamed videos, particularly over mobile/wireless networks. The algorithm adopts the no-reference method, and enables real-time measurement of video quality at any point in the content production and delivery chain using any given video. The technologies developed include no-reference methods for measuring picture freeze, picture loss, and blockiness. The developed system (where the software has not been optimized for speed) is able to process video of CIF size (352×288 pixels) at more than 30 fps on a Pentium-IV 3GHz computer. The experimental results show that the proposed video quality analysis system gives good accuracy for picture freeze, picture loss, and blocking detections.

This paper derives the formulae for defocus blur parameter from a single image, based upon the line spread function (LSF). To achieve high accuracy and robustness, the over determining strategies are adopted: 1) a number of LSFs on one edge are extracted; 2) more edges in the images are used. The trust-region method is then employed to obtain the optimal estimation of blur parameter. The experimental results have demonstrated the effectiveness of the proposed method. It can be used for blind image quality evaluation in vision-based applications.

The type and extent of blur affect image quality and therefore its evaluation. This paper presents an accurate method for blur identification and parameter estimation from one image without a priori knowledge. The key idea of the proposed method is to perform Fourier transform of logarithm spectrum to detect the periodic blur pattern in cepstrum domain instead of spectral nulls. Accordingly, the estimation of blur parameters is more accurate and robust to noise. The experimental results validate the accuracy of the proposed method.

This paper proposes a new full-reference objective metric for image quality assessment. The reference and distorted images are decomposed into a number of wavelet subbands, in which mean curvatures and perceived error of the wavelet coefficients of two images are computed and integrated to give overall quality index. Taking structural similarity and error visibility into account, the new method can achieve high consistency with subjective evaluation compared with other metrics. Experimental results have shown the effectiveness of the proposed metric.

In this paper, we present our work on developing perceptual quality metrics for low and very low bit-rate videos. At current stage, we've finished a number of subjective viewing experiments, and several video content-based models are built based on the experiments to measure the perceptual quality of frame-dropped and blurred video sequences. Experimental results show that the models can achieve very good performances.

The Region-of-interest (ROI) based video coding within Scalable Video Coding (SVC) can be implemented by making use of Type 2 Flexible Macroblock Ordering (FMO), which marks independent rectangle regions/slices inside a frame by their top-left and bottom-right coordinates. By employing the proposed scheme, a displaying frame can be separated into several independent regions that are assigned with different Signal-Noise-Ratio (SNR), Spatial and Temporal quality. The scheme can be used to ensure the achievement of high sub-jecitve quality or to fulfill some special functionalities. Owning to the fact that the frame is separated into independent regions, and the regions are assigned with big quality differences, false edge (blockiness) may appear around the ROI boundaries, which cannot be automatically removed by the in-loop filters. The annoyance of such artifact depends on the local visual context, thus a new adaptive deblocking filter is proposed in this paper. The filter includes two steps: first, the complexity of the blocks around ROI boundaries is measured; and different filtering modes and smoothing abilities are then selected accordingly to reduce the annoyance of the blockiness. Experimental results showed that coding quality is improved by the proposed filter in low bitrate Coarse Granular Scalability (CESB) video.

This paper proposes a method for assessing blur caused by defocus from one image. The essential idea is to estimate the point spread function (PSF) from the line spread function (LSF), whereas the LSF is constructed from edge information. The procedure includes edge detection and localization, pixel interpolation, LSF determination and PSF extraction. This approach is independent of image contents, and the algorithm has fast speed as it works in spatial domain without complex Fourier transform or iterative computation. The experimental results validate the proposed method. It can be used for blind image quality evaluation in vision-based applications.

This paper presents a novel illumination normalization approach for face recognition under varying lighting conditions. In the proposed approach, a discrete cosine transform (DCT) is employed to compensate for illumination variations in the logarithm domain. Since illumination variations mainly lie in the low-frequency band, an appropriate number of DCT coefficients are truncated to minimize variations under different lighting conditions. Experimental results on the Yale B database and CMU PIE database show that the proposed approach improves the performance significantly for the face images with large illumination variations. Moreover, the advantage of our approach is that it does not require any modeling steps and can be easily implemented in a real-time face recognition system.

In this paper, an efficient method for high-speed face recognition based on the discrete cosine transform (DCT), the Fisher's linear discriminant (FLD) and radial basis function (RBF) neural networks is presented. First, the dimensionality of the original face image is reduced by using the DCT and the large area illumination variations are alleviated by discarding the first few low-frequency DCT coefficients. Next, the truncated DCT coefficient vectors are clustered using the proposed clustering algorithm. This process makes the subsequent FLD more efficient. After implementing the FLD, the most discriminating and invariant facial features are maintained and the training samples are clustered well. As a consequence, further parameter estimation for the RBF neural networks is fulfilled easily which facilitates fast training in the RBF neural networks. Simulation results show that the proposed system achieves excellent performance with high training and recognition speed, high recognition rate as well as very good illumination robustness.

This paper presents a novel illumination normalization approach for face recognition under varying lighting conditions. First, we demonstrate that illumination compensation can be efficiently implemented in the logarithm domain. In the proposed approach, discrete cosine transform (DCT) is employed to compensate for illumination variations in the logarithm domain. Since illumination variations mainly lie in the low-frequency band, an appropriate number of DCT coefficients are truncated to reduce the variations under different lighting conditions. The salient feature of our approach is that it does not need any training or modelling step and can be easily implemented with high speed.

Color encoded phase-stepping light projection method is a new and promising technique for 3D shape modeling However, the 3D model acquired is often smeared by large error. The main cause of the error is color coupling amongst the three primary colors RGB. In this paper, we first analyzed the color-coupling problem. It is found that there is a strong coupling between G and R element. The coupling between R and G are proportional to the phase interval between them and the overall intensity of image. Second, we proposed an adaptive phase stepping method to alleviate the color coupling errors efficiently and improve accuracy effectively. An algorithm corresponding to a specific paradigm with R-G-B phase step set to 0-45-180 is given and is applied to measure different objects. Experimental results demonstrate the effectiveness of the method.

A general and efficient design approach using a radial basis function (RBF) neural classifier to cope with small training sets of high dimension, which is a problem frequently encountered in face recognition, is presented. In order to avoid overfitting and reduce the computational burden, face features are first extracted by the principal component analysis (PCA) method. Then, the resulting features are further processed by the Fisher's linear discriminant (FLD) technique to acquire lower-dimensional discriminant patterns. A novel paradigm is proposed whereby data information is encapsulated in determining the structure and initial parameters of the RBF neural classifier before learning takes place. A hybrid learning algorithm is used to train the RBF neural networks so that the dimension of the search space is drastically reduced in the gradient paradigm. Simulation results conducted on the ORL database show that the system achieves excellent performance both in terms of error rates of classification and learning efficiency.

A fast approach for automatically generating fuzzy rules from sample patterns using generalized dynamic fuzzy neural networks (GD-FNNs) is presented. The GD-FNN is built based on ellipsoidal basis functions and functionally is equivalent to a Takagi-Sugeno-Kang fuzzy system. The salient characteristics of the GD-FNN are: (1) structure identification and parameters estimation are performed automatically and simultaneously without partitioning input space and selecting initial parameters a priori; (2) fuzzy rules can be recruited or deleted dynamically; (3) fuzzy rules can be generated quickly without resorting to the backpropagation (BP) iteration learning, a common approach adopted by many existing methods. The GD-FNN is employed in a wide range of applications ranging from static function approximation and nonlinear system identification to time-varying drug delivery system and multilink robot control. Simulation results demonstrate that a compact and high-performance fuzzy rule-base can be constructed. Comprehensive comparisons with other latest approaches show that the proposed approach is superior in terms of learning efficiency and performance.

In this paper, an architecture of dynamic fuzzy neural networks (D-FNN) implementing Takagi-Sugeno-Kang (TSK) fuzzy systems based on extended radial basis function (RBF) neural networks is proposed. A novel learning algorithm based on D-FNN is also presented. The salient characteristics of the algorithm are: 1) hierarchical on-line self-organizing learning is used; 2) neurons can be recruited or deleted dynamically according to their significance to the system's performance; and 3) fast learning speed can be achieved. Simulation studies and comprehensive comparisons with some other learning algorithms demonstrate that a more compact structure with higher performance can be achieved by the proposed approach.