Symbolic Regression of Integer Sequences (SRIS) aims to discover precise mathematical formulas from integer sequences. The neural machine translation-based method of SRIS trains the model using randomly generated data, and directly utilizes the trained model for inference on target sequences. However, the method often fails to effectively generalize to the target sequence, since the randomly generated data can not adequately cover the distributions of target data, i.e., there are distribution differences between them. In this work, we propose a progressive self-learning (PSL) method to explicitly capture sequence-formula distributions of the target domain. Specifically, a source domain dataset is generated by incorporating initial terms of the target domain to reduce the sequence distribution gap between the source domain and the target domain. Meanwhile, a self-learning loop strategy is adopted to improve the ability of the model to capture the sequence-formula distribution of the target domain. In this strategy, a neural machine translation model is used to learn the mappings from sequences to formulas in an end-to-end fashion. Then, this model is employed to explore candidate formulas of the target sequence using beam search. After verifying these candidate formula correctness, some of them are retained as training data for the next learning. Experimental results on OEIS datasets demonstrate that the proposed method surpasses current state-of-the-art methods in accuracy, and also discovers new formulas.

In this paper, we propose a dual multi-modal feature fusion osteosarcoma evaluation network to address the problem of efficient utilization of osteosarcoma medical images for tumor necrosis rate assessment. Firstly, we propose a dual evaluation network to utilize dual input information. Then, a structure called the multi-modal information fusion module is incorporated into the network to focus on key information and enhance the information flow between network layers. Based on this framework, we propose a loss function called osteosarcoma generalization loss to solve the problem of data imbalance in osteosarcoma images. We conducted extensive experiments to evaluate the proposed method. The results of comparative ablation experiments show that our newly proposed method improves both the evaluation effectiveness and performance compared to existing methods.

Due to the scarcity of annotated samples, the diversity between support set and query set becomes the main obstacle for few shot semantic segmentation. Most existing prototype-based approaches only exploit the prototype from the support feature and ignore the information from the query sample, failing to remove this obstacle.In this paper, we proposes a dual prototype network (DPNet) to dispose of few shot semantic segmentation from a new perspective. Along with the prototype extracted from the support set, we propose to build the pseudo-prototype based on foreground features in the query image. To achieve this goal, the cycle comparison module is developed to select reliable foreground features and generate the pseudo-prototype with them. Then, a prototype interaction module is utilized to integrate the information of the prototype and the pseudo-prototype based on their underlying correlation. Finally, a multi-scale fusion module is introduced to capture contextual information during the dense comparison between prototype (pseudo-prototype) and query feature. Extensive experiments conducted on two benchmarks demonstrate that our method exceeds previous state-of-the-arts with a sizable margin, verifying the effectiveness of the proposed method.

Siamese-based trackers currently are the dominant tracking paradigm due to the balance between speed and performance. However, it is prone to drift and tracking failure when the environment is complex and similar objects interfere. While the Siamese-based trackers perform the correlation operation, the responses of the target object and background appear in different channels, i.e., the feature spaces of the target object and background have some orthogonality. However, when meeting background clutters and similar objects interfere, this orthogonality becomes weaker and the wrong classification contribution of the object and the background reduces the stability of the learned similarity function, leading to many misclassified pixels in the heatmaps. In this work, we proposed a SiamORPN to solve the above issues. It is incorporated at two levels: an Orthogonal Region Proposal Network (ORPN) and an Adaptive Pixel-wise Aggregation (APA) module. Specifically, for ORPN, the orthogonality between the object and the background maximizes the inter-class inertia. Moreover, the ORPN introduces the orthogonal module to enhance this orthogonality. For APA, it introduces two lightweight networks to predict the weights of all pixels in different heatmaps and the weights of all pixels in different regression offsets. Experiments on challenging benchmarks, including OTB2015, VOT2016, VOT2018, GOT-10k test set, UAV123, LaSOT, and TrackingNet, demonstrate the proposed SiamORPN outperforms many SOTA trackers and achieves leading performance. The inference speed at GTX1080Ti can reach about 32 FPS, meeting the real-time requirements.

Few shot segmentation is a newly-developing and challenging computer vision task which is only provided with few labeled samples of the novel class. Some recent works on this problem focus more on how to design an effective comparison module but ignore how to extract the features passed to compare. In this paper we propose a novel model named LTAF-Net for few-shot segmentation. This model aims to adaptively recalibrate the extracted features which could boost the accuracy of dense comparison between support features and query features. Besides an additional prediction refinement module is designed to refine the initial mask. Meanwhile this method can apply to k-shot setting without developing a new specialized architecture and achieve competitive performance. Experiments on PASCAL-5iand FSS-1000 strongly prove the effectiveness of the proposed model. Our model outperforms the second-best method 1.4% in 1-shot and 0.76% in 5-shot respectively in PASCAL-5i.

Deep learning has achieved huge success in the field of artificial intelligence, but the performance heavily depends on labeled data. Few-shot learning aims to make a model rapidly adapt to unseen classes with few labeled samples after training on a base dataset, and this is useful for tasks lacking labeled data such as medical image processing. Considering that the core problem of few-shot learning is the lack of samples, a straightforward solution to this issue is data augmentation. This paper proposes a generative model (VI-Net) based on a cosine-classifier baseline. Specifically, we construct a framework to learn to define a generating space for each category in the latent space based on few support samples. In this way, new feature vectors can be generated to help make the decision boundary of classifier sharper during the fine-tuning process. To evaluate the effectiveness of our proposed approach, we perform comparative experiments and ablation studies on mini-ImageNet and CUB. Experimental results show that VI-Net does improve performance compared with the baseline and obtains the state-of-the-art result among other augmentation-based methods.

Clustering is a crucial but challenging task in pattern analysis and machine learning. Existing methods often ignore the combination between representation learning and clustering. To tackle this problem, we reconsider the clustering task from its definition to develop Deep Self-Evolution Clustering (DSEC) to jointly learn representations and cluster data. For this purpose, the clustering task is recast as a binary pairwise-classification problem to estimate whether pairwise patterns are similar. Specifically, similarities between pairwise patterns are defined by the dot product between indicator features which are generated by a deep neural network (DNN). To learn informative representations for clustering, clustering constraints are imposed on the indicator features to represent specific concepts with specific representations. Since the ground-truth similarities are unavailable in clustering, an alternating iterative algorithm called Self-Evolution Clustering Training (SECT) is presented to select similar and dissimilar pairwise patterns and to train the DNN alternately. Consequently, the indicator features tend to be one-hot vectors and the patterns can be clustered by locating the largest response of the learned indicator features. Extensive experiments strongly evidence that DSEC outperforms current models on twelve popular image, text and audio datasets consistently.

Convolutional neural networks (CNNs) provide a dramatically powerful class of models, but are subject to traditional convolution that can merely aggregate permutation-ordered and dimension-equal local inputs. It causes that CNNs are allowed to only manage signals on Euclidean or grid-like domains (e.g., images), not ones on non-Euclidean or graph domains (e.g., traffic networks). To eliminate this limitation, we develop a local-aggregation function, a sharable nonlinear operation, to aggregate permutation-unordered and dimension-unequal local inputs on non-Euclidean domains. In the context of the function approximation theory, the local-aggregation function is parameterized with a group of orthonormal polynomials in an effective and efficient manner. By replacing the traditional convolution in CNNs with the parameterized local-aggregation function, Local-Aggregation Graph Networks (LAGNs) are readily established, which enable to fit nonlinear functions without activation functions and can be expediently trained with the standard back-propagation. Extensive experiments on various datasets strongly demonstrate the effectiveness and efficiency of LAGNs, leading to superior performance on numerous pattern recognition and machine learning tasks, including text categorization, molecular activity detection, taxi flow prediction, and image classification.

The image captioning is to describe an image with natural language as human, which has benefited from the advances in deep neural network and achieved substantial progress in performance. However, the perspective of human description to scene has not been fully considered in this task recently. Actually, the human description to scene is tightly related to the endogenous knowledge and the exogenous salient objects simultaneously, which implies that the content in the description is confined to the known salient objects. Inspired by this observation, this paper proposes a novel framework, which explicitly applies the known salient objects in image captioning. Under this framework, the known salient objects are served as the themes to guide the description generation. According to the property of the known salient object, a theme is composed of two components: its endogenous concept (what) and the exogenous spatial attention feature (where). Specifically, the prediction of each word is dominated by the concept and spatial attention feature of the corresponding theme in the process of caption prediction. Moreover, we introduce a novel learning method of Distinctive Learning (DL) to get more specificity of generated captions like human descriptions. It formulates two constraints in the theme learning process to encourage distinctiveness between different images. Particularly, reinforcement learning is introduced into the framework to address the exposure bias problem between the training and the testing modes. Extensive experiments on the COCO and Flickr30K datasets achieve superior results when compared with the state-of-the-art methods.

Xinyu Xiao, Lingfeng Wang, Bin Fan, Shinming Xiang, Chunhong Pan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Encoder-decoder models have been widely used in image captioning, and most of them are designed via single long short term memory (LSTM). The capacity of single-layer network, whose encoder and decoder are integrated together, is limited for such a complex task of image captioning. Moreover, how to effectively increase the “vertical depth” of encoder-decoder remains to be solved. To deal with these problems, a novel deep hierarchical encoder-decoder network is proposed for image captioning, where a deep hierarchical structure is explored to separate the functions of encoder and decoder. This model is capable of efficiently exerting the representation capacity of deep networks to fuse high level semantics of vision and language in generating captions. Specifically, visual representations in top levels of abstraction are simultaneously considered, and each of these levels is associated to one LSTM. The bottom-most LSTM is applied as the encoder of textual inputs. The application of the middle layer in encoder-decoder is to enhance the decoding ability of top-most LSTM. Furthermore, depending on the introduction of semantic enhancement module of image feature and distribution combine module of text feature, variants of architectures of our model are constructed to explore the impacts and mutual interactions among the visual representation, textual representations, and the output of the middle LSTM layer. Particularly, the framework is training under a reinforcement learning method to address the exposure bias problem between the training and the testing by the policy gradient optimization. Qualitative analyses indicate the process that our model “translates” image to sentence and further visualization presents the evolution of the hidden states from different hierarchical LSTMs over time. Extensive experiments demonstrate that our model outperforms current state-of-the-art models on three benchmark datasets: Flickr8K, Flickr30K, and MSCOCO. On both image captioning and retrieval tasks, our method achieves the best results. On MSCOCO captioning Leaderboard, our method also achieves superior performance.

Action recognition plays a fundamental role in computer vision and video analysis. Nevertheless, extracting effective spatial-temporal features remains a challenging task. This paper proposes three simple but effective weakly semantic guided modules (SGMs) for both environment-constrained and cross-domain action recognition. The SGMs are composed of total 3-D convolution and element-wise gated operations; thus, they are efficient and easy to implement. The semantic guidance is obtained in a weakly supervised manner, in which each video clip is labeled with only an action class instead of pixel-level semantics. Benefitting from the semantic guidance, the network [called semantic guided network (SGN)] can focus on the salient parts of the video clips. Consequently, the redundant information can be reduced and the model is more robust to noise. Besides, benefitting from the intrinsic property of SGMs, SGN is totally end-to-end trainable. Quantities of experiments on both environment-constrained (e.g., Penn, HMDB-51, and UCF101) and cross-domain (e.g., ODAR) action recognition datasets demonstrate its effectiveness. Specifically, SGN gets improvements of 3.7%, 2.1%, and 5.2% for Penn, HMDB-51, and UCF-101 than the baseline ResNet3D, respectively, and SGN ranked third place in the ODAR 2017 challenge.

One major difficulty in medical image segmentation is intensity inhomogeneity, which manifests itself with a slow intensity variation over the whole image domain. Recently, a local binary fitting (LBF) model has been proposed to solve this problem within level set segmentation framework. However, the LBF model has two main problems, i.e., high computational cost and sensitivity to initialization. By analyzing the LBF model, we find that the most computational part is the calculation of two cluster images, which need to be updated in each iteration during the evolution of level set function. With this observation in mind, we propose a novel two-scale filtering (TSF) model, in which the two cluster images can be pre-calculated before evolution. Additionally, we implicitly utilize order constraint to restrict the order of two cluster images. As a result, the proposed TSF model is less sensitive to initialization. Extensive experiments on real medical images illustrate the desirable performances, as compared with the state-of-the-art models.

Convolutional Neural Networks(CNNs) have achieved great success for object recognition in still images. However, CNNs can't make evident improvement for action recognition in videos, one reason is that many current network architectures are relatively shallow compared with deep models in image domain, and the other reason is that CNNs can't capture effective long-term motion information from videos. Encouraged by the good performance of Residual Network-s(ResNets) for training extremely deep models, and Long-term Recurrent Convolutional Networks(LSTMs) for dealing with tasks involving sequences, we presented an action recognition method based on a two-stream architecture, with 2D ResNets with LSTMs in one stream and designed 3D ResNets with LSTMs in the other stream, which can combine appearance and motion information better. Especially, our proposed method first learns spatiotemporal features of videos through the Residual networks, then models complex temporal dynamics by the Long-term Recurrent Convolutional networks, and with a softmax layer on the top of two streams, the final classification results can be predicted by fusing scores of each stream with weights on score distribution. Furthermore, for better reducing the influence of redundant background information in videos for recognition results, we also applied a center extraction method to generate central regions of videos instead of an entire video into a visual representation. On two video action benchmarks of UCF101 and HMDB51, our method achieved promising performance compared with state-of-the-art.

Deep learning has been successfully applied to single image super-resolution problem due to its high data fitting ability. However, the trending of deeper layers and wider receptive field to acquire better performance brings high computation complexity and serious information vanishing. To address this problem, we proposed a new Reconstructed DenseNets model for super-resolution. The basic idea behind Reconstructed DenseNets is to improve the recent DenseNets model by modifying the two core modules, dense blocks and transition blocks, so that the Reconstructed DenseNets can emphasize the quality of data reconstruction. Specifically, on the one hand, the batch normalization layers in dense blocks is ignored to overcome the data shift risk. One the other hand, the pooling layers in transition blocks is also ignored to ensure the ability to reconstruct. Based on the above two improvements, the new DenseNets is named as Reconstructed DenseNets. Extensive experiments evaluate the effectiveness of our model, showing the outperforming of the state-of-the-art approaches.

Convolutional neural networks (CNNs) are inherently subject to invariable filters that can only aggregate local inputs with the same topological structures. It causes that CNNs are allowed to manage data with Euclidean or grid-like structures (e.g., images), not ones with non-Euclidean or graph structures (e.g., traffic networks). To broaden the reach of CNNs, we develop structure-aware convolution to eliminate the invariance, yielding a unified mechanism of dealing with both Euclidean and non-Euclidean structured data. Technically, filters in the structure-aware convolution are generalized to univariate functions, which are capable of aggregating local inputs with diverse topological structures. Since infinite parameters are required to determine a univariate function, we parameterize these filters with numbered learnable parameters in the context of the function approximation theory. By replacing the classical convolution in CNNs with the structure-aware convolution, Structure-Aware Convolutional Neural Networks (SACNNs) are readily established. Extensive experiments on eleven datasets strongly evidence that SACNNs outperform current models on various machine learning tasks, including image classification and clustering, text categorization, skeleton-based action recognition, molecular activity detection, and taxi flow prediction.

Autoencoder, which learns latent representations of samples in an unsupervised manner, has great potential in computer vision and signal processing. However, the diversity of samples makes learning a component autoencoder remaining a challenging task. This letter proposes a novel Self-Paced AutoEncoder (SPAE) for unsupervised feature extraction. The motivation behind this letter is to take samples gradually from simple to complex into consideration during training, which is similar to the mechanism of knowledge acquisition for humans. Under the unsupervised learning framework constructed on the autoencoder infrastructure, our SPAE first learns a weak autoencoder via samples with small losses and, then, elevates itself to a relatively strong autoencoder through samples with large losses. Then, the SPAE is generalized to a temporal domain, resulting to temporal SPAE (TSPAE), where the temporal information is explored and exploited to improve the performance. Typically, a TSPAE is capable of compressing temporal sequences into temporal-independent data. Experiments on the image classification and action recognition demonstrate the effectiveness of SPAE and TSPAE.

In this brief, we propose a new groupwise retargeted least squares regression (GReLSR) model for multicategory classification. The main motivation behind GReLSR is to utilize an additional regularization to restrict the translation values of ReLSR, so that they should be similar within same class. By analyzing the regression targets of ReLSR, we propose a new formulation of ReLSR, where the translation values are expressed explicitly. On the basis of the new formulation, discriminative least-squares regression can be regarded as a special case of ReLSR with zero translation values. Moreover, a groupwise constraint is added to ReLSR to form the new GReLSR model. Extensive experiments on various machine leaning data sets illustrate that our method outperforms the current state-of-the-art approaches.

The goal of no-reference image quality assessment (NR-IQA) is to estimate human perceived image quality without access to either reference image or prior knowledge about distortion type. Previous approaches for this problem are typically based on a regression framework that maps the image features directly to a quality score. In contrast, psychological evidence shows that humans prefer to evaluate visual quality with qualitative descriptions, e.g., using a five-grade ordinal scale: “excellent”, “good”, “fair”, “poor” and “bad”. Based on this observation, we propose a vector regression model that predicts five belief scores rather than a single quality score. The belief scores are designed to indicate the confidences of the test image being assigned with these five quality grades. In addition, with the purpose of more extensive applications, a saliency-based pooling strategy is presented to convert the predicted confidences into objective quality scores. Extensive experiments performed on two benchmark datasets demonstrate that our approach achieves state-of-the-art performance and shows great generalization ability.

Discriminative least squares regression (DLSR) is a simple yet effective method for multi-class classification. One problem of DLSR is that it is lack of robustness to outliers. In order to tackle this difficulty, in this paper, we propose a novel Robust DLSR (RoDLSR) model. The core idea behind RoDLSR is to find and further ignore the outliers among the support vector set. Specifically, we modify the regression targets of outliers by adding an additional item. As a result, the range of regression residuals can be controlled within predefined threshold. Extensive experiments evaluate the effectiveness of RoDLSR, especially on the corrupted databases.

Image clustering is a crucial but challenging task in machine learning and computer vision. Existing methods often ignore the combination between feature learning and clustering. To tackle this problem, we propose Deep Adaptive Clustering (DAC) that recasts the clustering problem into a binary pairwise-classification framework to judge whether pairs of images belong to the same clusters. In DAC, the similarities are calculated as the cosine distance between label features of images which are generated by a deep convolutional network (ConvNet). By introducing a constraint into DAC, the learned label features tend to be one-hot vectors that can be utilized for clustering images. The main challenge is that the ground-truth similarities are unknown in image clustering. We handle this issue by presenting an alternating iterative Adaptive Learning algorithm where each iteration alternately selects labeled samples and trains the ConvNet. Conclusively, images are automatically clustered based on the label features. Experimental results show that DAC achieves state-of-the-art performance on five popular datasets, e.g., yielding 97.75% clustering accuracy on MNIST, 52.18% on CIFAR-10 and 46.99% on STL-10.

Semantic segmentation of indoor scene images has a wide range of applications. However, due to a large number of classes and uneven distribution in indoor scenes, mislabels are often made when facing small objects or boundary regions. Technically, contextual information may benefit for segmentation results, but has not yet been exploited sufficiently. In this paper, we propose a learnable contextual regularization model for enhancing the semantic segmentation results of color indoor scene images. This regularization model is combined with a deep convolutional segmentation network without significantly increasing the number of additional parameters. Our model, derived from the inherent contextual regularization on the indoor scene objects, benefits much from the learnable constraint layers bridging the lower layers and the higher layers in the deep convolutional network. The constraint layers are further integrated with a weighted L1-norm based contextual regularization between the neighboring pixels of RGB values to improve the segmentation results. Experimental results on NYUDv2 indoor scene dataset demonstrate the effectiveness and efficiency of the proposed method.

Deep learning techniques have been successfully applied in single image super-resolution (SR). Recently, researches have shown that increasing the depth of network can significantly improve SR performance. Very deep networks for SR achieved a large improvement than former methods. However, simply increasing depths basically introduce more parameters and this lead to cumbersome computational cost. In this paper, we present a general and effective method to accelerate very deep networks for single image SR. Our method is based on dilated convolution operation, which support exponential expansion of the receptive field without increasing filter size. With the help of dilated convolution, shallow networks can achieve large receptive field and exploit contextual information in an efficient way. Based on a very deep network, we propose a 12 layers dilated convolutional network for SR (DCNSR). While accelerating 2x speed, our shallow network achieves better performance than original deep networks and shows state-of-the-art reconstructed results.

Semantic labeling for the very high resolution (VHR) image of urban areas is challenging, because of many complex manmade objects with different materials and fine-structured objects located together. Under the framework of convolutional neural networks (CNNs), this paper proposes a novel end-to-end network for semantic labeling. Specifically, our network not only improves the labeling accuracy of complex manmade objects by aggregating multiple context semantics with a cascaded architecture, but also refines fine-structured objects by utilizing the low-level detail in shallow layers of CNNs with a hierarchical pyramid structure. Throughout the network, a dedicated residual correction scheme is employed to amend the latent fitting residual. As a result of these specific components, the whole model works in a global-to-local and coarse-to-fine manner. Experimental results show that our network outperforms the state-of-the-art methods on the large-scale ISPRS Vaihingen 2D Semantic Labeling Challenge dataset.

This paper presents a novel Linear Intrinsic Component (LIC) model for simultaneous estimation of bias field and segmentation of magnetic resonance (MR) images with the intensity inhomogene-ity. The core of LIC model is linear transformation, which is derived from Taylor expansion of non-linear model. Due to the linear transformation, observed image can be decomposed into four components, namely, true image, which characterizes a physical property of the tissues, multiplicative and additive bias fields, which result in intensity inhomogeneity, and Gaussian noises. Based on sub-space constraint on two bias fields, and piecewise smoothness restriction on the true image, we can performing the task of joint bias field estimation and image segmentation. To model the complex noises subject to non-gaussian distribution, we further extend LIC model by introducing the non-gaussian noise term, and propose the Non-Gaussian LIC (NGLIC) model. By adopting L1regularization to our solution, the NGLIC model can be effectively solved by iterative soft-thresholding approach. Both LIC and NGLIC models are evaluated on a lot of MR simulated images downloaded from Brain-Web and real images, showing the superiority to the state-of-the-art approach on both segmentation and bias field correction results.

Extracting spatial-temporal descriptors is a challenging task for video-based human action recognition. We decouple the 3D volume of video frames directly into a cascaded temporal spatial domain via a new convolutional architecture. The motivation behind this design is to achieve deep nonlinear feature representations with reduced network parameters. First, a 1D temporal network with shared parameters is first constructed to map the video sequences along the time axis into feature maps in temporal domain. These feature maps are then organized into channels like those of RGB image (named as Motion Image here for abbreviation), which is desired to preserve both temporal and spatial information. Second, the Motion Image is regarded as the input of the latter cascaded 2D spatial network. With the combination of the 1D temporal network and the 2D spatial network together, the size of whole network parameters is largely reduced. Benefiting from the Motion Image, our network is an end-to-end system for the task of action recognition, which can be trained with the classical algorithm of back propagation. Quantities of comparative experiments on two benchmark datasets demonstrate the effectiveness of our new architecture.

Fine-structured object segmentation is a challenging problem in object segmentation community. There are mainly two difficulties that can seriously degrade the segmentation quality: 1) insufficient interactions on fine structures due to the high demand of time and manual efforts, and 2) shrinking bias that discourages long object boundaries. To address these two issues, we develop a novel method within the graph cut framework. First, the commonly used operation of scribbling or dragging bounding boxes is replaced by loosely drawing a few rectangles, thus the interaction burden is largely reduced. Second, an edge-guided graph cut model is proposed to mitigate shrinking bias. This model enforces connectivity of fine structures by adjusting the weighting between neighboring pixels. Finally, the segmentation task is formulated as an optimization problem, which can be optimized effectively and efficiently. Comparative experimental results demonstrate the effectiveness of our method.

In this brief, we propose a new margin scalable discriminative least squares regression (MSDLSR) model for multicategory classification. The main motivation behind the MSDLSR is to explicitly control the margin of DLSR model. We first prove that the DLSR is a relaxation of the traditional L2-support vector machine. Based on this fact, we further provide a theorem on the margin of DLSR. With this theorem, we add an explicit constraint on DLSR to restrict the number of zeros of dragging values, so as to control the margin of DLSR. The new model is called MSDLSR. Theoretically, we analyze the determination of the margin and support vectors of MSDLSR. Extensive experiments illustrate that our method outperforms the current state-of-the-art approaches on various machine leaning and real-world data sets.

Urban building reconstruction is an important step for urban digitization and realisticvisualization. In this paper, we propose a novel automatic method to recover urban building geometry from 3D point clouds. The proposed method is suitable for buildings composed of planar polygons and aligned with the gravity direction, which are quite common in the city. Our key observation is that the building shapes are usually piecewise constant along the gravity direction and determined by several dominant shapes. Based on this observation, we formulate building reconstruction as an energy minimization problem under the Markov Random Field (MRF) framework. Specifically, point clouds are first cutinto a sequence of slices along the gravity direction. Then, floorplans are reconstructed by extracting boundaries of these slices, among which dominant floorplans are extracted and propagated to other floors via MRF. To guarantee correct propagation, a new distance measurement for floorplans is designed, which first encodes floorplans into strings and then calculates distances between their corresponding strings. Additionally, an image based editing method is also proposed to recover detailed window structures. Experimental results on both synthetic and real data sets have validated the effectiveness of our method.

Region-based level set methods have been widely used for image segmentation. Among them, the method based on local binary fitting (LBF) model is an efficient one. Unfortunately, LBF model is sensitive to initial contour. To overcome this disadvantage, we propose two explicit order models, i.e., the global order preserving and local order smoothness models. The global order preserving model ensures that the binary fitting values have the same order globally, while the local order smoothness model requires that these orders are smooth locally. With these two models, our segmentation results are not sensitive to initializations. Experimental results on synthetic and real images show desirable performances of our method, as compared with the state-of-the-art approaches.

In this paper, we present a novel method for the challenging task of fine-structured (FS) object segmentation. This task is formulated as a label propagation problem on an affinity graph. To enhance the completeness and connectivity of the FS objects, we introduce a novel neighborhood system combining both local and nonlocal connections, together with a robust scheme for edge weight calculation. Additionally, region cost is incorporated into the energy function to further maintain the connectivity of fine parts where the propagation is hard to reach. An appealing advantage of the proposed method is that the energy minimization has a closed-form solution and global optimum is guaranteed. Comparative experimental results on three datasets demonstrate the effectiveness of the proposed method.

Generating object proposals before object detection has become a common way. In this paper, we present a novel method to measure the objectness of bounding boxes using edges. The contours play an important role in object localization and detection. The number of edges that are close to the boundary of a box has strong relationship with the likelihood of the box covering an object. In our method, we adopt a two-step scheme to generate object proposals. In the first step, we count the number of contours close to the box, where we use the proposed “Tile Algorithm” to wipe off the inner edges of a box. In the second step we re-rank the object proposals with a linear SVM classifier across all aspect-ratios for calibration. Experiments on the VOC2007 dataset show that we achieve 96.47% object detection rate with 1000 proposals.

In this paper, we propose a new visual tracking method via the manifold regularized local structured sparse representation model under the particle filtering tracking framework. First, in order to tackle the difficulties of partial occlusion and illumination variation, the local structured sparse representation model is incorporated by exploiting both partial and spatial information of the target. Second, the manifold regularization is used to ensure that neighboring particles should share similar representation coefficients, so that these particles can cooperate with each other. Extensive experiments are performed on various video sequences, showing improvement over the state-of-the-art approaches.

In this paper, we propose a level-set based segmentation method for medical images with intensity inhomogeneity. Maximum a Posteriori estimation is adopted to combine image segmentation and bias field correction into a unified framework. Within this framework, both contour prior and bias field prior can be fully used. In order to restrict bias field, we introduce an adaptive regularization. Based on this new adaptive regularization, the bias field is estimated more smooth and the input medical image with intensity inhomogeneity is recovered more clearly. Especially, the estimated bias field of our method introduces less structure information obtained from input image. Experimental results on both synthetic and real images show the advantages of our method in both segmentation and bias field correction accuracies as compared with the state-of-the-art approaches.

We propose to integrate spectral-spatial feature extraction and tensor discriminant analysis for hyperspectral image classification. First, we apply remarkable spectral-spatial feature extraction approaches in the hyperspectral cube to extract a feature tensor for each pixel. Then, based on class label information, local tensor discriminant analysis is used to remove redundant information for subsequent classification procedure. The approach not only extracts sufficient spectral-spatial features from original hyperspectral images but also gets better feature representation owing to tensor framework. Comparative results on two benchmarks demonstrate the effectiveness of our method.

Sparse representation-(or sparse coding)-based classification has been successfully applied to face recognition. However, it can become problematic in the presence of illumination variations or occlusions. In this paper, we propose a Manifold Regularized Local Sparse Representation (MRLSR) model to address such difficulties. The key idea behind the MRLSR method is that all coding vectors in sparse representation should be group sparse, which means holding the two properties of both individual sparsity and local similarity. As a consequence, the face recognition rate can be considerably improved. The MRLSR model is optimized by the modified homotopy algorithm, which keeps stable under different choices of the weighting parameter. Extensive experiments are performed on various face databases, which contain illumination variations and occlusions. We show that the proposed method outperforms the state-of-the-art approaches and provides the highest recognition rate.

This brief presents a framework of retargeted least squares regression (ReLSR) for multicategory classification. The core idea is to directly learn the regression targets from data other than using the traditional zero-one matrix as regression targets. The learned target matrix can guarantee a large margin constraint for the requirement of correct classification for each data point. Compared with the traditional least squares regression (LSR) and a recently proposed discriminative LSR models, ReLSR is much more accurate in measuring the classification error of the regression model. Furthermore, ReLSR is a single and compact model, hence there is no need to train two-class (binary) machines that are independent of each other. The convex optimization problem of ReLSR is solved elegantly and efficiently with an alternating procedure including regression and retargeting as substeps. The experimental evaluation over a range of databases identifies the validity of our method.

Repetition extraction plays an important role in facade image analysis. In this paper, this task is handled within the graph cut based image segmentation framework. To model the repetitions, generalized translation symmetry (GTS) is introduced to enable aperiodic repetition layouts. More importantly, GTS is explicitly formulated in terms of matrix multiplication. That is, GTS is viewed as the product of a repetitive pattern and two block matrices. These two block matrices are employed to represent the vertical and horizontal symmetry respectively. On this basis, repetition extraction is formulated as a GTS constrained energy minimization problem. An alternatively optimization algorithm based on graph cut and dynamic programming is finally developed to solve the problem. Experimental results demonstrate the validity of our method.

In this paper, we propose a method for joint 2D segmentation and 2D-3D pose tracking. First, we define a novel energy functional which considers the discrimination between statistical appearance models and the coherence among neighboring pixels simultaneously. And then, a particle filter-like stochastic optimization technique is adopted to solve the energy functional, so that a preferable initial value can be provided for the subsequent damped Newton optimization method. Furthermore, an occlusion-aware updating strategy is utilized for appearance models, which can easily increase the foreground learning rate. As a result, our method is more suitable for the video sequences with occlusion. Experimental results highlight excellent performance on challenging synthetic and real-world sequences as compared with the state-of-the-art approaches.

Classification of hyper spectral images is an important issue in remote sensing image processing systems. Hyper spectral images have advantages in pixel-wise classification owing to the high spectral resolution. However, the pixel-wise classification result often introduces the salt-and-pepper appearance because of the complex noise produced by atmosphere and instrument. An effective way to overcome this phenomenon is to resort to the spatial information. This paper proposes a method to solve the above problem by using spatial similarity information. First, in order to avoid the effect of noisy pixels and mixed pixels, reliable seeds are selected in local windows according to the agreement between the central pixel and its spatial neighbors. Then, the information of the reliable seeds is propagated to their spatial neighbors by a graph Laplacian. Specifically, the graph Laplacian is designed to propagate information among spatial neighbors with close similarity relationship so that some small or long thin objects are identified. Through the seed selection and local reliable information propagation, the problem of noisy labels is solved elegantly. Experiments on three real hyper spectral data sets with different spatial resolution, spectral resolution and land covers demonstrate the effectiveness of our method.

It remains a challenging task to track an object robustly due to factors such as pose variation, illumination change, occlusion, and background clutter. In the past decades, a number of researchers have been attracted to tackling these difficulties, and they proposed many effective methods. Among them, sparse representation-based tracking method is a promising. While much success has been demonstrated, there are several issues that still need to be addressed. First, the introduction to trivial occlusion templates brings a high computational cost of this method. Second, the utilization of raw template object representation makes this method difficult to adopt sophisticated object features. To solve these problems, we consider the sparse representation problem in a kernel space and propose a kernel sparse representation (KSR)-based tracking algorithm. Under the kernel representation, it is not necessary to introduce trivial occlusion templates in order to reduce the computational cost. Furthermore, multikernel fusion allows our method to use multiple sophisticated object features, such as spatial color histogram and spatial gradient-orientation histogram, and let these features complement each other during the tracking process. Comparative experiments on challenging scenes demonstrate that our KSR-based tracking algorithm outperforms the state-of-the-art approaches in tracking accuracy.

In this paper, we present a fast image upsampling method within a two-scale framework to ensure the sharp construction of upsampled image for both large-scale edges and small-scale structures. In our approach, the low-frequency image is recovered via a novel sharpness preserving interpolation technique based on a well-constructed displacement field, which is estimated by a cross-resolution sharpness preserving model. Within this model, the distances of pixels on edges are preserved, which enables the recovery of sharp edges in the high-resolution result. Likewise, local high-frequency structures are reconstructed via a sharpness preserving reconstruction algorithm. Extensive experiments show that our method outperforms current state-of-the-art approaches, based on quantitative and qualitative evaluations, as well as perceptual evaluation by a user study. Moreover, our approach is very fast so as to be practical for real applications.

Frame structure estimation from line segments is an important yet challenging problem in understanding indoor scenes. In practice, line segment extraction can be affected by occlusions, illumination variations, and weak object boundaries. To address this problem, an approach for frame structure recovery based on line segment refinement and voting is proposed. We refined line segments by the revising, connecting, and adding operations. We then propose an iterative voting mechanism for selecting refined line segments, where a cross ratio constraint is enforced to build crab-like models. Our algorithm outperforms state-of-the-art approaches, especially when considering complex indoor scenes.

Super-resolution from a single image plays an important role in many computer vision systems. However, it is still a challenging task, especially in preserving local edge structures. To construct high-resolution images while preserving the sharp edges, an effective edge-directed super-resolution method is presented in this paper. An adaptive self-interpolation algorithm is first proposed to estimate a sharp high-resolution gradient field directly from the input low-resolution image. The obtained high-resolution gradient is then regarded as a gradient constraint or an edge-preserving constraint to reconstruct the high-resolution image. Extensive results have shown both qualitatively and quantitatively that the proposed method can produce convincing super-resolution images containing complex and sharp features, as compared with the other state-of-the-art super-resolution algorithms.

Object tracking plays an important role in many intelligent transportation systems. Unfortunately, it remains a challenging task due to factors such as occlusion and target-appearance variation. In this paper, we present a new tracking algorithm to tackle the difficulties caused by these two factors. First, considering the target-appearance variation, we introduce the local-background-weighted histogram (LBWH) to describe the target. In our LBWH, the local background is treated as the context of the target representation. Compared with traditional descriptors, the LBWH is more robust to the variability or the clutter of the potential background. Second, to deal with the occlusion case, a new forward-backward mean-shift (FBMS) algorithm is proposed by incorporating a forward-backward evaluation scheme, in which the tracking result is evaluated by the forward-backward error. Extensive experiments on various scenarios have demonstrated that our tracking algorithm outperforms the state-of-the-art approaches in tracking accuracy.

In this paper, we propose a novel multi-resolution background sub traction method. We adopt coarse to fine strategy, which is the essence the multi-resolution scheme, to obtain the foreground mask. The rough mask is first gained relied on the Single Gaussian Model, which holds minor computation cost. Then, the slightly accuracy mask is calculated by the Saliency-based Extraction Model, which contains high accuracy and stability. Finally, Contour-based Refining Model is used to refine the mask edge. Our algorithm is evaluated against several video sequences, and experimental results show that the proposed method is suitable for various scenes and is appealing with respect to robustness.

In this paper, we propose a new Mean-shift algorithm to tackle some tracking difficulties, such as background clutter and partial occlusion. First, we compare all Mean-shift-like tracking algorithms, and indicate that the main difference among them is weight calculation. Then, a new fusion strategy is proposed to unify all weight calculation methods into a framework. Based on this framework, we propose a novel weight calculation method, which takes the candidate model into consideration as well as incorporates the local background. Extensive experiments are conducted to evaluate the proposed approach. Comparative experimental results indicate that the tracking accuracy is improved as compared with the state-of-the-arts.

This paper presents a novel local region-based level set model for image segmentation. In each local region, we define a locally weighted least squares energy to fit a linear classification function. The local energy is then integrated over the entire image domain to form an energy functional in terms of level set function. The energy minimization is achieved by level set evolution and estimation of parameters of the locally linear function in an iterative process. By introducing the locally linear functions to separate background and foreground in local regions, our model not only ensures the accuracy of the segmentation results, but also be very robust to initialization. Experiments are reported to demonstrate the effectiveness and efficiency of our model.