We introduced StyleEntity, a zero-shot image manipulation model that utilizes named entities as proxies during its training phase. This strategy enables our model to manipulate images using unseen textual descriptions during inference, all within a single training phase. Additionally, we proposed an inference technique termed Prompt Ensemble Latent Averaging (PELA). PELA averages the manipulation directions derived from various named entities during inference, effectively eliminating the noise directions, thus achieving stable manipulation. In our experiments, StyleEntity exhibited superior performance in a zero-shot setting compared to other methods. The code, model weights, and datasets are available at https://github.com/feng-zhida/StyleEntity.

Image inpainting is a challenging task to recover the missing visual information of an image in a visually plausible way. To tackle this challenge, this paper proposes a new image in-painting approach that employs the edge structure knowledge to hierarchically guide image completion. The hierarchical guidance consists of the following two key components. First, a new edge structure aware fusion is proposed to encourage information fusion between image feature maps and edge structure feature maps hierarchically via a channel-wise attention mechanism. Second, a novel edge structure aware loss is proposed, with the help of the conventional image loss together, to reconstruct the refined image via hierarchically supervising side outputs from the image completion network. Extensive experimental results are provided to demonstrate that the proposed approach reconstructs images with better quantitative quality and better qualitative performance using two benchmark image datasets.

The success of single image super-resolution reconstruction (SR) relies on a refined mapping from low-resolution (LR) examples to high-resolution (HR) signals. However, the relation is sometimes chaos, especially in a deep SR network. We try to improve the mapping interpretability in two folds: i) The variable local dense blocks (VLDB) are suggested to match receptive fields in different depths of a residual dense network (RDN), with each VLDB a dyadic increment of layer numbers than its predecessor. ii) Based on VLDBs, a dense in dense (DID) network is created. It substitutes nodes in a regular RDN with super nodes, i.e. VLDBs; and formulates a joint learning by flexible hierarchical feature scaling, reusing and long-short term aggregating. VLDBs deal with feature underfitting occurred when a big receptive field meets a fixed-depth dense block, and the DID network provides a relative complete feature dictionary to preserve details for feature shift, dilating and grouping in high dimension image reconstruction. To demonstrate the validness of DID structure, detail experiments are performed on the benchmark datasets Set5, Set14, B100 and Urban100, where the accuracy PSNR and the visual perceptive SSIM are superior to most state-of-the-art methods. Besides, due to the depth adaption of VLDBs and its nesting in generalized RDN, DID network is converged easily and gradient explosion or disappearance are alleviated even when the network deepens.

In weakly-supervised temporal action detection, only the category labels of action instances could be used for training. Without any temporal boundary annotations in an untrimmed video, it is a huge challenge to propose the starter and the end of an action (named as "the proposa"). In this paper, we try to focus on the temporal differences of visual contents which may change significantly when motion occurs and carry the information of temporal boundaries. First, a motion-propagating feature vector (MPV) is designed to describe different kinds of temporal differences in a video. Second, we propose a temporal difference perceptron (TDP). According to the MPV, TDP computes an attention to each video snippet to infer action areas. However, the TDP only concentrates on the most discriminate action areas and the attentions fail in slow actions. To tackle these, an unbalanced action-background self-supervised loss (USSL) function is custom to tune the distribution of activated change areas. With the unbalanced constraint, all forward changes and some backward change action regions are activated. And the self-supervised attention labels force the model to widen the gap between background and action in feature space, contributing to boundaries inferring. Our method achieves comparable performance with the state-of-arts, which outperforms STPN, an algorithm without temporal difference message, more than 2% mAP on THUMOS14, and is 20.8% (mAP@tIou=0.5) higher than UntrimmedNets on ActivityNet v1.2 which used temporal difference information roughly.

A key challenge of saliency computation in foggy images is how to effectively detect salient objects which are less visible. The primary cause may lie in the fact that the light scattering through fog particles reduces image contrast. In this paper, we propose a frequency-spatial fusion saliency computational model based on discrete stationary wavelet transform (DSWT). The input image is firstly transformed into HSV color space, and the amplitude spectrum of each color channel is adjusted to generate the frequency domain saliency map. Then, the local-global superpixel contrast is measured to obtain the spatial domain saliency map. The DSWT is finally utilized to fuse the frequency-spatial cues. Experimental results indicate that the proposed model can efficiently reduce the influence of light scattering through fog particles, and can achieve the best performance in foggy images comparing to 16 state-of-the-art saliency models.

Diabetic retinopathy (DR) is a common retinal disease that leads to blindness. For diagnosis purposes, DR image grading aims to provide automatic DR grade classification, which is not addressed in conventional research methods of binary DR image classification. Small objects in the eye images, like lesions and microaneurysms, are essential to DR grading in medical imaging, but they could easily be influenced by other objects. To address these challenges, we propose a new deep learning architecture, called BiRA-Net, which combines the attention model for feature extraction and bilinear model for fine-grained classification. Furthermore, in considering the distance between different grades of different DR categories, we propose a new loss function, called grading loss, which leads to improved training convergence of the proposed approach. Experimental results are provided to demonstrate the superior performance of the proposed approach.

This paper studies the automatic classification of abnormal mutant and normal fishes by analyzing their movements recorded in the video. Motivated by the observation that mutant fishes have muscle disorders so that their bodies cannot bend sufficiently to swim normally, a kinematics-driven movement change detection approach is proposed to automatically segment the recorded video into different video segments. Furthermore, a new feature extraction method, called local kinematics shape pattern (LKSP), is proposed in this paper to provide discriminative spatiotemporal kinematics measurements of fish body movements. The histogram of the proposed LKSP features is incorporated into a motion classification approach to identify whether the fish is normal or a mutant. The experiments are conducted using the real-world recorded videos to demonstrate the superior performance of the proposed approach.

Temporal action localization is a very important yet challenging task, since video in real applications is usually long, unconstrained and untrimmed, containing multiple action instances plus video content of complex background scenes. The task recognizes not only the action class but also the start frame and the end frame of each action instance. To address this issue, we propose a novel temporal dynamic pooling (TDP) network to determine the temporal boundaries of an action instance and to select key frames of a video through a sequential prediction of importance score, which is obtained by score prediction network realized based on residual learning. By pooling only a few key frames containing discriminative information, we could obtain the salient pooled vector for action recognition. We conduct experiments on a challenging temporal action localization dataset, THUMOS 2014. When setting Intersection-over-Union (IoU) threshold to 0.5 in evaluation, we significantly achieve superior performances compared with the most state-of-the-art methods by increasing mAP from 22.7% to 29.8%. Meanwhile, our TDP network demonstrates a very high efficiency with the ability to process 550 fps on a single CPU.

The objective of image contrast enhancement is to improve the contrast level of images, which are degraded during image acquisition. Image contrast enhancement is considered as an optimization problem in this paper and the artificial bee colony (ABC) algorithm is utilized to find the optimal solution for this optimization problem. The contribution of the proposed approach is two-fold. First, in view of that the fitness function is indispensable to evaluate the quality of the enhanced image, a new multiple objective fitness function is proposed in this paper. Second, the image transformation function is critical to generate new pixel intensities for the enhanced image from the original input image; more importantly, it guides the searching movements of the artificial bees. For that, a parametric image transformation function is utilized in this paper so that only the optimal parameters used in the transformation function need to be searched by the ABC algorithm. This is in contrast to that the whole space of image intensity levels is used in the conventional ABC-based image enhancement approaches. Extensive experiments are conducted to demonstrate that the proposed approach outperforms conventional image contrast enhancement approaches to achieve both better visual image quality and higher objective performance measures.

Biomedical studies show that mutant genes in transgenic mutant fishes can lead to muscle disorders so that their swimming capabilities are affected. This paper studies the automatic detection of abnormal mutant fishes by analyzing their movements in the video. To differentiate between normal fish and mutant fish, a new feature extraction method, called spatiotemporal local kinematics pattern (STLKP), is proposed in this paper to provide discriminative spatiotemporal kinematics measurements of fish body movements. Furthermore, the histogram of the proposed STLKP features is incorporated into a motion classification approach to identify whether the fish is normal or a mutant. A large collection of real-world recorded videos is used in experiments to demonstrate that the proposed approach outperforms the conventional approaches to provide more accurate abnormal motion detection performance.

Blind noisy assessment aims to evaluate the quality of the degraded noisy image/video without the need for the ground-truth image. To tackle this challenge, this paper proposes a new blind noise assessment approach based on the path information of the ants' movement. The proposed modified ant shortest path (MASP) algorithm uses the path information of the ant colony optimization (ACO). The contribution of the proposed approach is two-fold. First, the proposed approach utilizes a number of artificial ants to move on a 2-D graph for constructing the path information, and calculates the ants' movement path driven by the shortest path strategy. Second, several path statistics metrics are proposed to evaluate the image quality. Experimental results are provided to demonstrate that the proposed image quality assessment approach is effective for both benchmark image database and real-world noisy video.

Visual saliency is an important cue in human visual system, it can identify salient region in image. Image contrast has been utilized as an effective feature to detect the salient region. The conventional contrast measures utilize both spectral and spatial properties of image in many salient region detection methods. However, they only consider the local characteristics of image region, consequently, the global characteristics are neglected. This paper presented a new contrast measure by exploiting both local and global characteristics of image regions. Furthermore, the proposed measure is utilized to perform salient region detection in image. Experiments are conducted on the MSRA test database to compare the performance of the proposed approach with the state-of-the-art salient region detection algorithms.

This paper tackles the fish motion synthesis problem with the aim to increase the temporal resolution by generating a high frame rate video from a given video captured at a lower frame rate. The proposed approach reconstructs the virtual frames via estimating motion-vector fields that represent the correspondences between consecutive frames. To be more specific, the proposed approach utilizes two regularization functions: (i) an intra regularization function that is proposed to impose a local smoothness constraint on the estimated motion vector fields; and (ii) an inter regularization function that considers both intensity consistency and kinematic consistency between the reconstructed frame and its neighboring frames. Real-world fish motion video dataset is used to demonstrate the superior performance of the proposed approach in high temporal resolution fish motion synthesis, as well as its application in fish motion analysis.

This paper presents a novel despeckling algorithm that can be used to enhance image quality in medical ultrasound images. Firstly, the log-transformed images are transformed by dual-tree complex wavelet transform (DTCWT). And then, we use a non-Gaussian statistical model with an adaptive smoothing parameter for ideal image signal in the transformed domain. According to Bayesian theory, the MAP estimator is obtained with a proposed adaptive threshold which has better despeckling performance by exploiting the interscale properties of wavelet coefficients. The proposed approach results in significant speckle reduction and preserve details of ultrasound images at the same time while the introduced distortions are not noticeable.

Blind noisy image quality assessment aims to evaluate the quality of the degraded noisy image without the need for the ground truth image. To tackle this challenge, this paper proposes an image quality assessment approach using block homogeneity. The contribution of the proposed approach is two-fold. First, a block-based homogeneity measure is proposed to estimate the statistics (e.g., variance) of the noise incurred in the image, based on adaptively selected homogeneous image regions. Second, an image quality assessment approach is proposed by exploiting the above-mentioned estimated noise variance, along with the visual masking effect of the human visual system. Experimental results are provided to demonstrate that the proposed image noise estimation approach yields superior accuracy and stability performance to that of conventional approaches, and the proposed image quality assessment approach achieves consistent performance to that of human subjective evaluation.

This paper proposes an improved backtracking-based adaptive orthogonal matching pursuit (OMP) algorithm (IBAOMP) and applies it to ultrasonic RF echoes signals. We first model the residual signal of an ultrasonic image with a Gaussian-like distribution to analyse the relationship between BAOMP thresholding and sparsity. Then we improve the BAOMP algorithm by a variable atoms selecting thresholding to avoid possible wrong match and residual increase in upper iterations of BAOMP. The proposed IBAOMP triggers the calibration of enrolled atom candidates to avoid the quasi-periodic mismatch of residual signals and upgrades the atommatch rate. We compare IBAOMP with other greedy algorithms for the biomedical ultrasonic image and 1D sparse signals. Numerical experiments demonstrate that the proposed IBAOMP algorithm can obviously prevent the residual rebounds for the biomedical ultrasonic image reconstruction and increase the exact recovery probability by 26% in comparison to BAOMP for Gaussian sparse signal.

Atom selection is crucial to compressive sensing (CS) reconstruction by orthogonal matching pursuit (OMP), where the look-ahead (LA) OMP algorithm (LAOMP) evaluated final effects of all the LA atoms before they were included into a support set, certainly, a high computation burden has to be suffered. This paper modifies LAOMP method by two folds: 1) Regularization (R-LAOMP) is introduced to restrict the atom selection by similar small residuals, while mutual effects of new selected atoms are considered to alleviate the high computation costs. 2) Backtracking-based (LA-BOMP) atom pruning is employed to remove the most mismatching atoms in support sets to balance the accuracy and the random disturbance in optimization procedures. Accordingly this regularized forward atom evaluation combining backward atom deleting method (R-LA-BOMP) leads to a significant improvement in LAOMP, while a trade-off between performance and complexity is achieved. Experiments of the regularized atom selection and the backtracking pruning algorithms are performed on Gaussian sparse signals, 0-1 sparse signals and speech voices and the results are given.

The estimation of noise statistics is critical to optimize many computer vision algorithms. The main issue is how to identify the homogeneous image patches for estimating the noise statistics. The smallest variance used in the conventional approaches is not always a good measure of homogeneity of image patches. In addition, the conventional approaches neglect the fact that homogeneous image patches tend to cluster together due to local spatial smoothness in images. In view of this, a new image noise estimation approach is proposed in this letter. The proposed approach has two key components. First, a graphical representation is proposed to model the relationship among image patches. Second, the ant colony optimization (ACO) technique is used to automatically select a set of patches for estimating the noise statistics. To be more specific, the proposed approach guides the spatial movement of artificial ants towards homogeneous locations in the graph, by considering both global (i.e., clustering measure) properties and local (i.e., homogeneity measure) properties of patches. Experimental results are provided to justify that the proposed approach out-performs nine conventional approaches to provide more accurate noise statistics estimation.

In this paper we propose a novel corner detection algorithm using local adaptive thresholding and iterative approaching. First, a new metric is defined to measure the saliency of the corner responses within a local region. With this metric, the task to find corner points is transformed into a task to find those regions with high saliency of corner responses. Then an iterative strategy is adopted to gradually narrow down these regions and finally approaching the actual positions of true corners. We demonstrate that the proposed algorithm is equivalent to a local adaptive thresholding method and thus can effectively suppress edge points. Experiments show that both its false detection rate and miss detection rate are lower than those of Noble detector, and its average detection accuracy is higher than the latter by 3.58%.

Resolution enhancement for stereo imaging aims to use a pair of stereo images to reconstruct another pair of images with higher resolution. To tackle this problem, a Bayesian resolution enhancement approach is proposed in this paper. Since the prior image model is essential for solving the ill-posed numerical issues encountered in the image resolution enhancement, the proposed approach exploits a prior image model which considers both the spatial local smoothness constraint within each reconstructed high-resolution image and the disparity-compensated local smoothness constraint between the pair of reconstructed high-resolution images. Then the proposed prior image model is further incorporated into a Bayesian inference formulation to perform stochastic image reconstruction. Experiments are conducted to demonstrate the superior performance of the proposed approach.

The aim of multi-focus image fusion is to combine multiple images with different focuses for enhancing the perception of a scene. The critical issue in the design of multi-focus image fusion algorithms is to evaluate the local content information of the input images. Motivated by the observation that the marginal distribution of the wavelet coefficients is different for images with different focus levels, an image fusion approach using wavelet-domain statistics is proposed in this paper. The proposed approach exploits the spreading of the wavelet coefficients distribution to measure the degree of the image's blur. Furthermore, the wavelet coefficients distribution is evaluated using a locally-adaptive Laplacian mixture model. Extensive experiments are conducted using three sets of test images under three objective metrics to demonstrate the superior performance of the proposed approach.

Ant colony optimization (ACO) is an optimization algorithm inspired by the natural behavior of ant species that ants deposit pheromone on the ground for foraging. In this paper, ACO is introduced to tackle the image edge detection problem. The proposed ACO-based edge detection approach is able to establish a pheromone matrix that represents the edge information presented at each pixel position of the image, according to the movements of a number of ants which are dispatched to move on the image. Furthermore, the movements of these ants are driven by the local variation of the image’s intensity values. Experimental results are provided to demonstrate the superior performance of the proposed approach.

The super-resolution (SR) imaging is to overcome the inherent limitations of the image acquisition systems to produce high-resolution images from their low-resolution counterparts. In our recent work, the Markov chain Monte Carlo (MCMC) technique has been successfully developed and shown as a promising stochastic approach for addressing the SR problem. However, the MCMC SR approach requires substantial amounts of computational resources, for it not only needs to generate a huge number of samples, but also requires an exhaustive search for obtaining an optimal prior image model. To tackle the above computation challenge, Grid computing is introduced for tackling the SR problem in this paper. The computationally- intensive MCMC SR task is broke down into a set of independent and small sub-tasks, which are further distributed and implemented in the grid computing environment. Their respective results are finally assembled to produce a high-resolution image as the final result of the entire MCMC SR task. Experiments are conducted to show that grid computing can effectively accelerating the execution time of the MCMC SR algorithm.

In this paper, we consider the super-resolution image reconstruction problem. We propose a Markov chain Monte Carlo (MCMC) approach to find the maximum a posterior probability (MAP) estimation of the unknown high-resolution image. Firstly, Gaussian Markov random field (GMRF) is exploited for modeling the prior probability distribution of the unknown high-resolution image. Then, a MCMC technique (in particular, the Gibbs sampler) is introduced to generate samples from the posterior probability distribution to compute the MAP estimation of the unknown high-resolution image, which is obtained as the mean of the samples. Moreover, we derive a bound on the convergence time of the proposed MCMC approach. Finally, the experimental results are presented to verify the superior performance of the proposed approach and the validity of the proposed bound.

Super-resolution imaging is to overcome the inherent limitations of image acquisition to create high-resolution images from their low-resolution counterparts. In this paper, a novel state-space approach is proposed to incorporate the temporal correlations among the low-resolution observations into the framework of the Kalman filtering. The proposed approach exploits both the temporal correlations information among the high-resolution images and the temporal correlations information among the low-resolution images to improve the quality of the reconstructed high-resolution sequence. Experimental results show that the proposed framework is superior to bi-linear interpolation, bi-cubic spline interpolation and the conventional Kalman filter approach, due to the consideration of the temporal correlations among the low-resolution images.