Visual Question Answering (VQA) is a representative task of cross-modal reasoning where an image and a free-form question in natural language are presented and the correct answer needs to be determined using both visual and textual information. One of the key issues of VQA is to reason with semantic clues in the visual content under the guidance of the question. In this paper, we propose Scene Graph Convolutional Network (SceneGCN) to jointly reason the object properties and their semantic relations for the correct answer. The visual relationship is projected into a deep learned semantic space constrained by visual context and language priors. Based on comprehensive experiments on two challenging datasets: GQA and VQA 2.0, we demonstrate the effectiveness and interpretability of the new model.

In recent years, deep learning based video frame interpolation methods have shown impressive results in handling occlusion, blur and large motion. However, they are usually very heavy in terms of model size, and they hardly to be employed in i.e. mobile phones or other portable devices with limited computing power. To address the problem, we propose light-weighted Spatial Pyramid Frame Interpolation Network (SPFIN), a hierarchical network in a coarse-to-fine approach to reconstruct frames. At each pyramid level, we apply two light sub-networks to model optical flow and visibility mask instead of commonly used U-Net architecture. The flow and mask are up-sampled and optimized progressively. Finally, the intermediate frame is formed by linearly blending warped frames and masks. Experimental results on two benchmark problems show that our model has the smallest size, but better or comparable performance comparing to existing state-of-the art models.

Stock volatility prediction is a challenging task in time-series prediction according to the Efficient Market Hypothesis which supposes all the investors are rational. However, many theories have showed that stock markets are not efficient due to the effects of psychological and social factors. In this paper, we constructed self-attention networks (SAN) to quantify the impact on the volatility of Chinese stock market of social information, such as social opinion and social concern. Our SAN model can explore the relationships among features at different time steps more flexibly, and thus, explore stock historical information more effectively. Empirical results show the superiority of our model compared to other existing models on given stock data.

Semantic segmentation is one of the basic topics in computer vision, it aims to assign semantic labels to every pixel of an image. Unbalanced semantic label distribution could have a negative influence on segmentation accuracy. In this paper, we investigate using data augmentation approach to balance the semantic label distribution in order to improve segmentation performance. We propose using generative adversarial networks (GANs) to generate realistic images for improving the performance of semantic segmentation networks. Experimental results show that the proposed method can not only improve segmentation performance on those classes with low accuracy, but also obtain 1.3% to 2.1% increase in average segmentation accuracy. It shows that this augmentation method can boost the accuracy and be easily applicable to any other segmentation models.

The recent advances of deep learning in both computer vision (CV) and natural language processing (NLP) provide us a new way of understanding semantics, by which we can deal with more challenging tasks such as automatic description generation from natural images. In this challenge, the encoder-decoder framework has achieved promising performance when a convolutional neural network (CNN) is used as image encoder and a recurrent neural network (RNN) as decoder. In this paper, we introduce a sequential guiding network that guides the decoder during word generation. The new model is an extension of the encoder-decoder framework with attention that has an additional guiding long short-term memory (LSTM) and can be trained in an end-to-end manner by using image/descriptions pairs. We validate our approach by conducting extensive experiments on a benchmark dataset, i.e., MS COCO Captions. The proposed model achieves significant improvement comparing to the other state-of-the-art deep learning models.

Cardiovascular disease is the number one killer that threatens human health. Analysis of coronary vascular geometry plays an important role in the diagnosis of cardiovascular disease. In order to make reliable judgments for clinical usage, accurate and robust vascular segmentation methods are needed. We present an improved segmentation approach for dynamic coronary angiography sequences, which can obtain blood vessel trees in real time. We start with a pre-trained convolutional neural network (CNN) on ImageNet as the base network for initialization. We then adapt this network to the structure of one-shot video object segmentation (OSVOS) to obtain the parent network, which is trained on 50 X-ray angiography sequences for extracting blood vessels. Finally, the model is further fine-tuned using 10 specific coronary angiography sequences to achieve real-time vascular segmentation. Our method achieved the Dice index of 0.88, suggesting that the presented method can be potentially useful in developing a real-time segmentation method for coronary angiography sequences.

Carotid atherosclerosis is a primary cause of stroke, which is responsible for a majority of disabilities and deaths worldwide. Plaque inflammation and abundant microvasculature have been identified as important aspects contributing to plaque vulnerability that can be studied non-invasively with dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI). Due to the asymptomatic nature of vulnerable plaque, there is an unmet clinical need to identify and characterize these lesions before they rupture. We presented an automated computerized method based on kinetic measures to distinguish vulnerable from stable atherosclerotic plaques on DCE-MRI. Four classes of kinectic features, including pharmacokinetic, intensity kinetic, histogram kinetic, and textual kinetic features, were extracted for capturing the pathophysiologic changes in various aspects of plaque vascular structure and functionality in atherosclerosis. These features can reflect the local inflammatory processes and microvasculature changes appearing in plaque destabilization. Our method was evaluated on real clinical data and achieved the area under the curve of 0.95 using a combined feature set, suggesting a potential of this method applied to a computer-aided diagnosis system for an early detection of vulnerable plaques.

Single image super-resolution (SR) aims at generating a high-resolution (HR) image from one low-resolution (LR) input. In this paper, we focus on single image SR by using a confidence growing model based on an example-based super resolution approach. Compared to previous works that reconstruct high-resolution image in a raster scan order, the new proposed method reconstructs the patches using a new confidence measure. More confident reconstructions are propagated to neighboring areas by enforcing a smoothness constraint in selecting patches. We also adopt hierarchical clustering to construct a training set to speed up processing. Experimental results demonstrate that this simple method outperforms existing state-of-the-art algorithms on a the given benchmark SR test images.

To diagnose breast cancer (BCa), the number of mitotic cells present in tissue sections is an important parameter to examine and grade breast biopsy specimen. The differentiation of mitotic from non-mitotic cells in breast histopathological images is a crucial step for automatical mitosis detection. This work aims at improving the accuracy of mitosis classification by characterizing objects of interest (tissue cells) in wavelet based multi-resolution representations that better capture the statistical features having mitosis discrimination. A dual-tree complex wavelet transform (DT-CWT) is performed to decompose the image patches into multi-scale forms. Five commonly-used statistical features are extracted on each wavelet subband. Since both mitotic and non-mitotic cells appear as small objects with a large variety of shapes in the images, characterization of mitosis is a challenging problem. The inter-scale dependencies of wavelet coefficients allow extraction of important texture features within the cells that are more likely to appear at all different scales. The wavelet-based statistical features were evaluated on a dataset containing 327 mitotic and 406 non-mitotic cells via a support vector machine classifier in iterative cross-validation. The quantitative results showed that our DT-CWT based approach achieved superior classification performance with the accuracy of 87.94%, sensitivity of 86.80%, specificity of 89.89%, and the area under the curve (AUC) value of 0.94.

Saliency detection is one of the extraordinary capabilities of the human visual system (HVS). In this paper, we present a novel saliency detection model to capture visual selective attention of images. The new model does not require prior knowledge of salient regions as well as manual labeling. The mean shift segmentation algorithm and quaternion discrete cosine transform (QDCT) are used to generate a rough saliency map by integrating low-level features and spatial saliency information. In each segmented region, the color saliency is measured based on the probability of its occurrences in foreground and background defined by the rough saliency map. The experimental results on a widely used benchmark database demonstrated that the presented model achieves the best performance in terms of visual and quantitative evaluations compared to existing state-of-the-art saliency detection models.

Multifocus image fusion is an important research topic in the computer vision and image processing field. The optical lenses that are commonly used by imaging devices, such as auto-focus cameras, have a limiting focus range. Thus, only objects within the range of distances from the devices can be captured and recorded sharply while out-of-range objects become blur. In this paper, we present a novel image fusion scheme for combining two or multiple images with different focus points to generate an all-in-focus image. We formulate the problem of fusing multifocus images as choosing most significant features from a sparse matrix produced by a newly developed robust principal component analysis (RPCA) decomposition method to form a composite feature space. Thus, the salient features presented in sharp regions can be captured and integrated into a single representation. The sparse matrix is first divided into small blocks, and standard deviation is then calculated on each block as a selection criterion. To reduce blocking artifacts, a sliding window technique is utilized to smooth the transitions between blocks. The proposed fusion scheme has been demonstrated to successfully improve fusion quality in terms of visual and quantitative evaluations. The method is also able to effectively handle both grayscale and color images.

Contemporary video search and categorization are non-trivial tasks due to the massively increasing amount and content variety of videos. We put forward the study of visual saliency models in video. Such a model is employed to identify salient objects from the image background. Starting from the observation that motion information in video often attracts more human attention compared to static images, we devise a region contrast based saliency detection model using spatial-temporal cues (RCST). We introduce and study four saliency principles to realize the RCST. This generalizes the previous static image for saliency computational model to video. We conduct experiments on a publicly available video segmentation database where our method significantly outperforms seven state-of-the-art methods with respect to PR curve, ROC curve and visual comparison.

Color perception is one of the major cognitive abilities of human being. Color information is also one of the most important features in various computer vision tasks including object recognition, tracking, scene classification and so on. In this paper, we proposed a simple and effective method for learning color composition of objects from large annotated datasets. The new proposed model is based on a region-based bag-of-colors model and saliency detection. The effectiveness of the model is empirically verified on manually labelled datasets with single or multiple tags. The significance of this research is that the color information of an object can provide useful prior knowledge to help improving the existing computer vision models in image segmentation, object recognition and tracking.

A novel region-based image fusion framework based on multiscale image segmentation and statistical feature extraction is proposed. A dual-tree complex wavelet transform (DT-CWT) and a statistical region merging algorithm are used to produce a region map of the source images. The input images are partitioned into meaningful regions containing salient information via symmetric alpha-stable (S alphaS) distributions. The region features are then modeled using bivariate alpha-stable (B alphaS) distributions, and the statistical measure of similarity between corresponding regions of the source images is calculated as the Kullback-Leibler distance (KLD) between the estimated B alphaS models. Finally, a segmentation-driven approach is used to fuse the images, region by region, in the complex wavelet domain. A novel decision method is introduced by considering the local statistical properties within the regions, which significantly improves the reliability of the feature selection and fusion processes. Simulation results demonstrate that the bivariate alpha-stable model outperforms the univariate alpha-stable and generalized Gaussian densities by not only capturing the heavy-tailed behavior of the subband marginal distribution, but also the strong statistical dependencies between wavelet coefficients at different scales. The experiments show that our algorithm achieves better performance in comparison with previously proposed pixel and region-level fusion approaches in both subjective and objective evaluation tests.

A novel context enhancement technique is presented to automatically combine images of the same scene captured at different times or seasons. A unique characteristic of the algorithm is its ability to extract and maintain the meaningful information in the enhanced image while recovering the surrounding scene information by fusing the background image. The input images are first decomposed into multiresolution representations using the Dual-Tree Complex Wavelet Transform (DT-CWT) with the subband coefficients modelled as Cauchy random variables. Then, the convolution of Cauchy distributions is applied as a probabilistic prior to model the fused coefficients, and the weights used to combine the source images are optimised via Maximum Likelihood (ML) estimation. Finally, the importance map is produced to construct the composite approximation image. Experiments show that this new model significantly improves the reliability of the feature selection and enhances fusion process.

Compressive sensing (CS) has received a lot of interest due to its compression capability and lack of complexity on the sensor side. In this paper, we present a study of three sampling patterns and investigate their performance on CS reconstruction. We then propose a new image fusion algorithm in the compressive domain by using an improved sampling pattern. There are few studies regarding the applicability of CS to image fusion. The main purpose of this work is to explore the properties of compressive measurements through different sampling patterns and their potential use in image fusion. The study demonstrates that CS-based image fusion has a number of perceived advantages in comparison with image fusion in the multiresolution (MR) domain. The simulations show that the proposed CS-based image fusion algorithm provides promising results.

A novel multiscale image segmentation algorithm is presented, which is based on the dominant color and homogeneous texture features (HTF) that are adopted in the MPEG-7 standard. These features are efficiently combined to perform the automatic segmentation. First, the image is roughly segmented into textured and nontextured regions using Gabor decomposition. A multiscale segmentation is then applied to the resulting regions, according to the local texture feature. Finally, a precise boundary refinement procedure is employed to accurately determine the boundaries between textured and nontextured regions. A novel region merging algorithm is introduced with a simple and effective segment classification by using HTF to deal with the over-segmentation problem. Experiments show that our algorithm provides an improved performance compared with JSEG and a watershed algorithm.

This paper presents a new statistical image segmentation algorithm, in which the texture features are modeled by symmetric alpha-stable (SalphaS) distributions. These features are efficiently combined with the dominant color feature to perform automatic segmentation. First, the image is roughly segmented into textured and nontextured regions using the dual-tree complex wavelet transform (DT-CWT) with the sub-band coefficients modeled as SalphaS random variables. A mul-tiscale segmentation is then applied to the resulting regions, according to the local texture characteristics. Finally, a novel statistical region merging algorithm is introduced by measuring the Kullback-Leibler distance (KLD) between estimated SalphaS models for the neighboring segments. Experiments show that our algorithm achieves superior segmentation results in comparison with existing state-of-the-art image segmentation algorithms.