Brain Magnetic Resonance Images (MRIs) are commonly used for tumor diagnosis. Machine learning for brain tumor characterization often uses MRIs from many modalities (e.g., T1-MRI, Enhanced-T1-MRI, T2-MRI and FLAIR). This paper tackles two issues that may impact brain tumor characterization performance from deep learning: insufficiently large training dataset, and incomplete collection of MRIs from different modalities. We propose a novel pairwise generative adversarial network (GAN) architecture for generating synthetic brain MRIs in missing modalities by using existing MRIs in other modalities. By improving the training dataset, we aim to mitigate the overfitting and improve the deep learning performance. Main contributions of the paper include: (a) propose a pairwise generative adversarial network (GAN) for brain image augmentation via cross-modality image generation; (b) propose a training strategy to enhance the glioma classification performance, where GAN-augmented images are used for pre-training, followed by refined-training using real brain MRIs; (c) demonstrate the proposed method through tests and comparisons of glioma classifiers that are trained from mixing real and GAN synthetic data, as well as from real data only. Experiments were conducted on an open TCGA dataset, containing 167 subjects for classifying IDH genotypes (mutation or wild-type). Test results from two experimental settings have both provided supports to the proposed method, where glioma classification performance has consistently improved by using mixed real and augmented data (test accuracy 81.03%, with 2.57% improvement).

This paper addresses the issues of Alzheimer's disease (AD) characterization and detection from Magnetic Resonance Images (MRIs). Many existing AD detection methods use single-scale feature learning from brain scans. In this paper, we propose a multiscale deep learning architecture for learning AD features. The main contributions of the paper include: (a) propose a novel 3D multiscale CNN architecture for the dedicated task of AD detection; (b) propose a feature fusion and enhancement strategy for multiscale features; (c) empirical study on the impact of several settings, including two dataset partitioning approaches, and the use of multiscale and feature enhancement. Experiments were conducted on an open ADNI dataset (1198 brain scans from 337 subjects), test results have shown the effectiveness of the proposed method with test accuracy of 93.53%, 87.24% (best, average) on subject-separated dataset, and 99.44%, 98.80% (best, average) on random brain scan-partitioned dataset. Comparison with eight existing methods has provided further support to the proposed method.

Compared to conventional saliency detection by handcrafted features, deep convolutional neural networks (CNNs) recently have been successfully applied to saliency detection field with superior performance on locating salient objects. However, due to repeated sub-sampling operations inside CNNs such as pooling and convolution, many CNN-based saliency models fail to maintain fine-grained spatial details and boundary structures of objects. To remedy this issue, this paper proposes a novel end-to-end deep learning-based refinement model named Refinet, which is based on fully convolutional network augmented with segmentation hypotheses. Intermediate saliency maps that are edge-aware are computed from segmentation-based pooling and then feed to a two-tier fully convolutional network for effective fusion and refinement, leading to more precise object details and boundaries. In addition, the resolution of feature maps in the proposed Refinet is carefully designed to guarantee sufficient boundary clarity of the refined saliency output. Compared to widely employed dense conditional random field, Refinet is able to enhance coarse saliency maps generated by existing models with more accurate spatial details, and its effectiveness is demonstrated by experimental results on seven benchmark datasets.

This paper addresses issues of grading brain tumor, glioma, from Magnetic Resonance Images (MRIs). Although feature pyramid is shown to be useful to extract multi -scale features for object recognition, it is rarely explored in MRI images for glioma classification/grading. For glioma grading, existing deep learning methods often use convolutional neural networks (CNN s) to extract single-scale features without considering that the scales of brain tumor features vary depending on structure/shape, size, tissue smoothness, and locations. In this paper, we propose to incorporate the multi-scale feature learning into a deep convolutional network architecture, which extracts multi -scale semantic as well as fine features for glioma tumor grading. The main contributions of the paper are: (a) propose a novel 3D multi-scale convolutional network architecture for the dedicated task of glioma grading; (b) propose a novel feature fusion scheme that further refines multi -scale features generated from multi-scale convolutional layers; (c) propose a saliency -aware strategy to enhance tumor regions of MRIs. Experiments were conducted on an open dataset for classifying high/low grade gliomas. Performance on the test set using the proposed scheme has shown good results (with accuracy of 89.47%).

Salient object detection is aimed at detecting and segmenting objects that human eyes are most focused on when viewing a scene. Recently, conditional random field (CRF) is drawn renewed interest, and is exploited in this field. However, when utilizing a CRF with unary and pairwise potentials having essential parameters, most existing methods only employ manually designed parameters, or learn parameters partly for the unary potentials. Observing that the saliency estimation is a continuous labeling issue, this paper proposes a novel data-driven scheme based on a special CRF framework, the so-called continuous CRF (C-CRF), where parameters for both unary and pairwise potentials are jointly learned. The proposed C-CRF learning provides an optimal way to integrate various unary saliency features with pairwise cues to discover salient objects. To the best of our knowledge, the proposed scheme is the first to completely learn a C-CRF for saliency detection. In addition, we propose a novel formulation of pairwise potentials that enables learning weights for different spatial ranges on a superpixel graph. The proposed C-CRF learning-based saliency model is tested on 6 benchmark datasets and compared with 11 existing methods. Our results and comparisons have provided further support to the proposed method in terms of precision-recall and F-measure. Furthermore, incorporating existing saliency models with pairwise cues through the C-CRF are shown to provide marked boosting performance over individual models.

This paper addresses the problem of classifying activities of daily living in video. The proposed method uses a tree structure of two layers, where in each node of the tree there resides a Riemannian manifold that corresponds to different part-based covariance features. In the first layer, activities are classified according to the dynamics of upper body parts. In the second layer, activities are further classified according to the appearance of local image patches at hands in key frames, where the interacting objects are likely to be attached. The novelties of this paper include: (i) characterizing the motion of upper body parts by a covariance matrix of distances between each pair of key points and the orientations of lines that connect them; (ii) describing human-object interaction by the appearance of local regions around hands in key frames that are selected based on the proximity of hands to other key points; (iii) formulating a pairwise geodesics-based kernel for activity classification on Riemannian manifolds under the log-Euclidean metric. Experiments were conducted on a video dataset containing a total number of 426 video events (activities) from 4 classes. The proposed method is shown to be effective by achieving high classification accuracy (93.79% on average) and small false alarms (1.99% on average) overall, as well as for each individual class.

In this paper we address the issue of enhancing salient object detection through diffusion-based techniques. For reliably diffusing the energy from labeled seeds, we propose a novel graph-based diffusion scheme called affinity learning-based diffusion (ALD), which is based on learning full-range affinity between two arbitrary graph nodes. The method differs from the previous existing work where implicit diffusion was formulated as a ranking problem on a graph. In the proposed method, the affinity learning is achieved in a unified graph-based semi-supervised manner, whose outcome is leveraged for global propagation. By properly selecting an affinity learning model, the proposed ALD outperforms the ranking-based diffusion in terms of accurately detecting salient objects and enhancing the correct salient objects under a range of background scenarios. By utilizing the ALD, we propose an enhanced saliency detector that outperforms 7 recent state-of-the-art saliency models on 3 benchmark datasets.

This paper addresses issues in fall detection from RGB-D videos. The study focuses on measuring the dynamics of shape and motion of the target person, based on the observation that a fall usually causes drastic large shape deformation and physical movement. The main novelties include: (a) forming contours of target persons in depth images based on morphological skeleton; (b) extracting local dynamic shape and motion features from target contours; (c) encoding global shape and motion in HOG and HOGOF features from RGB images; (d) combining various shape and motion features for enhanced fall detection. Experiments have been conducted on an RGB-D video dataset for fall detection. Results show the effectiveness of the proposed method.

The establishment of robust target appearance model over time is an overriding concern in visual tracking. In this paper, we propose an inverse nonnegative matrix factorization (NMF) method for robust appearance modeling. Rather than using a linear combination of nonnegative basis vectors for each target image patch in conventional NMF, the proposed method is a reverse thought to conventional NMF tracker. It utilizes both the foreground and background information, and imposes a local coordinate constraint, where the basis matrix is sparse matrix from the linear combination of candidates with corresponding nonnegative coefficient vectors. Inverse NMF is used as a feature encoder, where the resulting coefficient vectors are fed into a SVM classifier for separating the target from the background. The proposed method is tested on several videos and compared with seven state-of-the-art methods. Our results have provided further support to the effectiveness and robustness of the proposed method.

Visual-based traffic sign recognition (TSR) requires first detecting and then classifying signs from captured images. In such a cascade system, classification accuracy is often affected by the detection results. This paper proposes a method for extracting a salient region of traffic sign within a detection window for more accurate sign representation and feature extraction, hence enhancing the performance of classification. In the proposed method, a superpixel-based distance map is firstly generated by applying a signed geodesic distance transform from a set of selected foreground and background seeds. An effective method for obtaining a final segmentation from the distance map is then proposed by incorporating the shape constraints of signs. Using these two steps, our method is able to automatically extract salient sign regions of different shapes. The proposed method is tested and validated in a complete TSR system. Test results show that the proposed method has led to a high classification accuracy (97.11%) on a large dataset containing street images. Comparing to the same TSR system without using saliency-segmented regions, the proposed method has yielded a marked performance improvement (about 12.84%). Future work will be on extending to more traffic sign categories and comparing with other benchmark methods.

Central nervous system dysfunction in infants may be manifested through inconsistent, rigid and abnormal limb movements. Detection and quantification of these movements in infants from videos are hence desirable for providing useful information to clinicians. This could lead to computer-aided diagnosis of dysfunctions where early treatment may improve infant development. In this paper, we propose a scheme for detecting and quantifying qualitative aspects of limb movement through multiple tracking and state space motion modeling on videos. The main novelties of the paper include: (a) An enhanced detection method for effectively detection small weak marker points from video; (b) Bayesian estimation and nearest neighbor searching for selecting new observation in individual tracker and for tracking marker trajectories on limbs; (c) A criterion for anomaly detection based on the frequency and duration of abrupt changes in limb movement, using window averaged prominent residual powers. The proposed method has been tested on videos of neonates, results show that the proposed method is promising for tracking and quantifying the movement of neonate limbs for helping medical diagnostics.

This paper addresses issues in fall detection from videos. Since it has been a broadly accepted intuition that a falling person usually undergoes large physical movement and displacement in a short time interval, the study is thus focused on measuring the intensity and temporal variation of pose change and body motion. The main novelties of this paper include: (a) characterizing pose/motion dynamics based on centroid velocity, head-to-centroid distance, histogram of oriented gradients and optical flow; (b) extracting compact features based on the mean and variance of pose/motion dynamics; (c) detecting human by combining depth information and background mixture models. Experiments have been conducted on an RGB-D video dataset for fall detection. Tests and evaluations show the effectiveness of the proposed method.

Recently the Normalized cut (Ncut) has been introduced to salient object detection [1, 2]. In this paper we validate that instead of proposing new detection models that leverage the Ncut, the previous geodesic saliency detection model which computes shortest paths on a graph can be adapted to eigenvectors of the Ncut to produce superior performance. Since the Ncut partitions a graph in a normalized energy minimization fashion, resulting eigenvectors contain decent cluster information that can group visual contents. Combining it with the existing geodesic saliency detection is conducive to highlighting salient objects uniformly, yielding to improved detection accuracy. Experiments by comparing with 12 existing methods on four benchmark datasets show the proposed method significantly outperforms the original geodesic saliency model and achieves comparable performance to state-of-the-art methods.

This paper addresses issues in fall detection from videos. The focus is on the analysis of human shapes which deform drastically in camera views while a person falls onto the ground. A novel approach is proposed that performs fall detection from an arbitrary view angle, via shape analysis on a unified Riemannian manifold for different camera views. The main novelties of this paper include: (a) representing dynamic shapes as points moving on a unit n-sphere, one of the simplest Riemannian manifolds; (b) characterizing the deformation of shapes by computing velocity statistics of their corresponding manifold points, based on geodesic distances on the manifold. Experiments have been conducted on two publicly available video datasets for fall detection. Test, evaluations and comparisons with 6 existing methods show the effectiveness of our proposed method.

Traffic sign recognition, including sign detection and classification, is essential for advanced driver assistance systems and autonomous vehicles. This paper introduces a novel machine learning-based sign recognition scheme. In the proposed scheme, detection and classification are realized through learning in a coarse-to-fine manner. Based on the observation that signs in the same category share some common attributes in appearance, the proposed scheme first distinguishes each individual sign category from the background in the coarse learning stage (i.e. sign detection) followed by distinguishing different sign classes within each category in the fine learning stage (i.e. sign classification). Both stages are realized through machine learning techniques. A complete recognition scheme is developed that is effective for simultaneously recognizing multiple categories of traffic signs. In addition, a novel saliency-based feature extraction method is proposed for sign classification. The method segments salient sign regions by leveraging the geodesic energy propagation. Compared with the conventional feature extraction, our method provides more reliable feature extraction from salient sign regions. The proposed scheme is tested and validated on two categories of Chinese traffic signs from Tencent street view. Evaluations on the test dataset show reasonably good performance, with an average of 97.5% true positive and 0.3% false positive on two categories of traffic signs.

Existing salient object detection models favor over-segmented regions upon which saliency is computed. Such local regions are less effective on representing object holistically and degrade emphasis of entire salient objects. As a result, the existing methods often fail to highlight an entire object in complex background. Toward better grouping of objects and background, in this paper, we consider graph cut, more specifically, the normalized graph cut (Ncut) for saliency detection. Since the Ncut partitions a graph in a normalized energy minimization fashion, resulting eigenvectors of the Ncut contain good cluster information that may group visual contents. Motivated by this, we directly induce saliency maps via eigenvectors of the Ncut, contributing to accurate saliency estimation of visual clusters. We implement the Ncut on a graph derived from a moderate number of superpixels. This graph captures both intrinsic color and edge information of image data. Starting from the superpixels, an adaptive multi-level region merging scheme is employed to seek such cluster information from Ncut eigenvectors. With developed saliency measures for each merged region, encouraging performance is obtained after across-level integration. Experiments by comparing with 13 existing methods on four benchmark datasets, including MSRA-1000, SOD, SED, and CSSD show the proposed method, Ncut saliency, results in uniform object enhancement and achieves comparable/better performance to the state-of-the-art methods.

The main drawback of conventional filtering based methods for small dim target (SDT) detection is they could not guarantee sufficient suppression ability towards trivial high frequency component which belongs to background, such as strong corners and edges. To overcome this bottleneck, this paper proposes an effective SDT detection algorithm by using local connectedness constraint. Our method provides direct control for target size, ensure high accuracy and could be easily embedded into the classical sliding-window based framework. The effectiveness of the proposed method is validated using images with cluttered background.

This paper addresses issues in video object tracking. We propose a novel method where tracking is regarded as a one-class classification problem of domain-shift objects. The proposed tracker is inspired by the fact that the positive samples can be bounded by a closed hypersphere generated by one-class support vector machines (SVM), leading to a solution for robust learning of target model online. The main novelties of the paper include: (a) represent the target model by a set of positive samples as a cluster of points on Riemannian manifolds; (b) perform online learning of target model as a dynamic cluster of points flowing on the manifold, in an alternate manner with tracking; (c) formulate geodesic-based kernel function for one-class SVM on Riemannian manifolds under the log-Euclidean metric. Experiments are conducted on several videos, results have provided support to the proposed method.

Many existing methods for salient object detection are performed by over-segmenting images into non-overlapping regions, which facilitate local/global color statistics for saliency computation. In this paper, we propose a new approach: spectral salient object detection, which is benefited from selected attributes of normalized cut, enabling better retaining of holistic salient objects as comparing to conventionally employed pre-segmentation techniques. The proposed saliency detection method recursively bi-partitions regions that render the lowest cut cost in each iteration, resulting in binary spanning tree structure. Each segmented region is then evaluated under criterion that fit Gestalt laws and statistical prior. Final result is obtained by integrating multiple intermediate saliency maps. Experimental results on three benchmark datasets demonstrate the effectiveness of the proposed method against 13 state-of-the-art approaches to salient object detection.

Recently many graph-based salient region/object detection methods have been developed. They are rather effective for still images. However, little attention has been paid to salient region detection in videos. This paper addresses salient region detection in videos. A unified approach towards graph construction for salient object detection in videos is proposed. The proposed method combines static appearance and motion cues to construct graph, enabling a direct extension of original graph-based salient region detection to video processing. To maintain coherence in both intra- and inter-frames, a spatial-temporal smoothing operation is proposed on a structured graph derived from consecutive frames. The effectiveness of the proposed method is tested and validated using seven videos from two video datasets.

Detecting pedestrians is a challenging and widely explored problem in computer vision. Many approaches rely on large quantities of manually labelled training data to learn a pedestrian classifier. To reduce the need for collecting and manually labelling real image training data, this paper investigates the possibility to use augmented images to train a pedestrian classifier. Augmented images are generated by rendering virtual pedestrians onto real image backgrounds. Classifiers learned from real or augmented training data are evaluated on real image test data from the widely used Daimler Mono Pedestrian benchmark data set. Results show that augmented training data generated from a single 200 frame image sequence reach 70 % average detection rate at one False Positives Per Image (FPPI), compared to 81 % for a classifier trained by a large-scale real data set. Results also show that complementing real training data with augmented data improves detection performance, compared to using real training data only.

Super-Resolution and High Dynamic Range image reconstruction are two different signal processing techniques that share in common that they utilize information from multiple observations of the same scene to enhance visual image quality. In this paper, both techniques are merged in a common model, and the focus is to solve the reconstruction problem in a suitable image domain, which relates to the perception of the Human Visual System. Simulated results are presented, including a comparison with a conventional method, demonstrating the benefits of the proposed approach, in this case avoiding some severe reconstruction artifacts.

This paper proposes a novel geodesic saliency propagation method where detected salient objects may be isolated from both the background and other clutter by adding global considerations in the detection process. The method transmits saliency energy from a coarse saliency map to all image parts rather than from image boundaries in conventional cases. The coarse saliency map is computed using the combination of global contrast and Harris convex hull. Superpixels from pre-segmented image are used as pre-processing to further enhance the efficiency. The proposed propagation is geodesic distance assisted and retains the local connectivity of objects. It is capable of rendering a uniform saliency map while suppressing the background, leading to salient objects being popped out. Experiments were conducted on a benchmark dataset, visual comparisons and performance evaluations with 9 existing methods have shown that the proposed method is robust and achieves the state-of-the-art performance.

This paper addresses the issue of classification of human activities in still images. We propose a novel method where part-based features focusing on human and object interaction are utilized for activity representation, and classification is designed on manifolds by exploiting underlying Riemannian geometry. The main contributions of the paper include: (a) represent human activity by appearance features from image patches containing hands, and by structural features formed from the distances between the torso and patch centers; (b) formulate SVM kernel function based on the geodesics on Riemannian manifolds under the log-Euclidean metric; (c) apply multi-class SVM classifier on the manifold under the one-against-all strategy. Experiments were conducted on a dataset containing 2750 images in 7 classes of activities from 10 subjects. Results have shown good performance (average classification rate of 95.83%, false positive rate of 0.71%). Comparisons with three other related classifiers provide further support to the proposed method.

This paper proposes a novel Bayesian online learning and tracking scheme for video objects on Grassmann manifolds. Although manifold visual object tracking is promising, large and fast nonplanar (or out-of-plane) pose changes and long-term partial occlusions of deformable objects in video remain a challenge that limits the tracking performance. The proposed method tackles these problems with the main novelties on: 1) online estimation of object appearances on Grassmann manifolds; 2) optimal criterion-based occlusion handling for online updating of object appearances; 3) a nonlinear dynamic model for both the appearance basis matrix and its velocity; and 4) Bayesian formulations, separately for the tracking process and the online learning process, that are realized by employing two particle filters: one is on the manifold for generating appearance particles and another on the linear space for generating affine box particles. Tracking and online updating are performed in an alternating fashion to mitigate the tracking drift. Experiments using the proposed tracker on videos captured by a single dynamic/static camera have shown robust tracking performance, particularly for scenarios when target objects contain significant nonplanar pose changes and long-term partial occlusions. Comparisons with eight existing state-of-the-art/most relevant manifold/nonmanifold trackers with evaluations have provided further support to the proposed scheme.

This paper discusses resolution enhancement of a set of images with varying exposure durations, having a high combined dynamic range. So far, little has been said in relation to the Human Visual System when it comes to Super-Resolution and High Dynamic Range fusion, unlike the case for traditional Super-Resolution where errors are measured with respect to human perception in the pixel domain. We propose a Super-Resolution method in the L*a*b* domain to bridge that gap and present some image reconstruction results.

This paper proposes a novel method for multi-view face pose classification through sequential learning and sensor fusion. The basic idea is to use face images observed in visual and thermal infrared (IR) bands, with the same sampling weight in a multi-class boosting structure. The main contribution of this paper is a multi-class AdaBoost classification framework where information obtained from visual and infrared bands interactively complement each other. This is achieved by learning weak hypothesis for visual and IR band independently and then fusing the optimized hypothesis sub-ensembles. In addition, an effective feature descriptor is introduced to thermal IR images. Experiments are conducted on a visual and thermal IR image dataset containing 4844 face images in 5 different poses. Results have shown significant increase in classification rate as compared with an existing multi-class AdaBoost algorithm SAMME trained on visual or infrared images alone, as well as a simple baseline classification-fusion algorithm.

This paper addresses online learning of reference object distribution in the context of two hybrid tracking schemes that combine the mean shift with local point feature correspondences, and the mean shift under the Bayesian framework, respectively. The reference object distribution is built up by a kernel-weighted color histogram. The main contributions of the proposed schemes includes: (a) an adaptive learning strategy that seeks to update the reference object distribution when the changes are caused by the intrinsic object dynamic with out partial occlusion/intersection; (b) novel dynamic maintenance of object feature points by exploring both foreground and background sets; (c) integration of adaptive appearance and local point features in joint object appearance similar ity and local point features correspondences-based tracker to improve; (d) integration of adaptive appearance in joint appearance similarity and particle filter tracker under the Bayesian framework to improve. Experimental results on a range of videos captured by a dynamic/stationary cam era demonstrate the effectiveness of the proposed schemes in terms of robustness to partial occlusions, tracking drifts and tightness and accuracy of tracked bounding box. Comparisons are also made with the two hybrid trackers together with 3 existing trackers.

We propose a novel visual tracking scheme that exploits both the geometrical structure of Grassmann manifold and piece-wise geodesics under a Bayesian framework. Two particle filters are alternatingly employed on the manifold. One is used for online updating the appearance subspace on the manifold using sliding-window observations, and the other is for tracking moving objects on the manifold based on the dynamic shape and appearance models. Main contributions of the paper include: (a) proposing an online manifold learning strategy by a particle filter, where a mixture of dynamic models is used for both the changes of manifold bases in the tangent plane and the piecewise geodesics on the manifold, (b) proposing a manifold object tracker by incorporating object shape in the tangent plane and the manifold prediction error of object appearance jointly in a particle filter framework. Experiments performed on videos containing significant object pose changes show very robust tracking results. The proposed scheme also shows better performance as comparing with three existing trackers in terms of tracking drift and the tightness and accuracy of tracked boxes.

This paper addresses issues in object tracking where videos contain complex scenarios. We propose a novel tracking scheme that jointly employs particle filters and multi-mode anisotropic mean shift. The tracker estimates the dynamic shape and appearance of objects, and also performs online learning of reference object. Several partition prototypes and fully tunable parameters are applied to the rectangular object bounding box for improving the estimates of shape and multiple appearance modes in the object. The main contributions of the proposed scheme include: 1) use a novel approach for online learning of reference object distributions; 2) use a five parameter set (2-D central location, width, height, and orientation) of rectangular bounding box as tunable variables in the joint tracking scheme; 3) derive the multi-mode anisotropic mean shift related to a partitioned rectangular bounding box and several partition prototypes; and 4) relate the bounding box parameter computation with the multi-mode mean shift estimates by combining eigen decomposition, geometry of subareas, and weighted average. This has led to more accurate and efficient tracking where only small number of particles (<;20) is required. Experiments have been conducted for a range of videos captured by a dynamic or stationary camera, where the target object may experience long-term partial occlusions, intersections with other objects with similar color distributions, deformable object accompanied with shape, pose or abrupt motion speed changes, and cluttered background. Comparisons with existing methods and performance evaluations are also performed. Test results have shown marked improvement of the proposed method in terms of robustness to occlusions, tracking drifts and tightness and accuracy of tracked bounding box. Limitations of the method are also mentioned.

This paper proposes a novel visual object tracking scheme, exploiting both local point feature correspondences and global object appearance using the anisotropic mean shift tracker. Using a RANSAC cost function incorporating the mean shift motion estimate, motion smoothness and complexity terms, an optimal feature point set for motion estimation is found even when a high proportion of outliers is presented. The tracker dynamically maintains sets of both foreground and background features, the latter providing information on object occlusions. The mean shift motion estimate is further used to guide the inclusion of new point features in the object model. Our experiments on videos containing long term partial occlusions, object intersections and cluttered or close color distributed background have shown more stable and robust tracking performance in comparison to three existing methods.

A novel visual object tracking scheme is proposed by using joint point feature correspondences and object appearance similarity. For point feature-based tracking, we propose a candidate tracker that simultaneously exploits two separate sets of point feature correspondences in the foreground and in the surrounding background, where background features are exploited for the indication of occlusions. Feature points in these two sets are then dynamically maintained. For object appearance-based tracking, we propose a candidate tracker based on an enhanced anisotropic mean shift with a fully tunable (five degrees of freedom) bounding box that is partially guided by the above feature point tracker. Both candidate trackers contain a reinitialization process to reset the tracker in order to prevent accumulated tracking error propagation in frames. In addition, a novel online learning method is introduced to the enhanced mean shift-based candidate tracker. The reference object distribution is updated in each time interval if there is an indication of stable and reliable tracking without background interferences. By dynamically updating the reference object model, tracking is further improved by using a more accurate object appearance similarity measure. An optimal selection criterion is applied to the final tracker based on the results of these candidate trackers. Experiments have been conducted on several videos containing a range of complex scenarios. To evaluate the performance, the proposed scheme is further evaluated using three objective criteria, and compared with two existing trackers. All our results have shown that the proposed scheme is very robust and has yielded a marked improvement in terms of tracking drift, tightness, and accuracy of tracked bounding boxes, especially for complex video scenarios containing long-term partial occlusions or intersections, deformation, or background clutter with similar color distributions to the foreground object.

NMR spectroscopy is a powerful technique used in protein research for comprehensive functional characterizations, e.g. structure determination at atomic resolution. Due to the molecular size (typically≫1000 atoms), protein NMR spectra contain a large number of signal frequencies. Resolving these requires high-dimensional spectroscopy. However, when the number of frequency exceeds three, conventional approaches start to demand unrealistic long experiment time, and the data analysis becomes challenging. In this paper we explore a combination of novel methods: Data from 5D NMR experiments are recorded as a series of 2D projections, which are then subjected to 2D subband filters and 2D LS-ESPRIT for estimation of signal parameters. Based on the relations established between 5D NMR signals and their 2D counterparts, projection spectroscopy allows to extract highly similar information as what would be found in conventional 5D NMR experiments; however, while the latter would require months of experiment time, the recording of all necessary projections can be accomplished within 1–2 days. Preliminary results show the efficiency of the method with respect to accuracy and resolution of the parameter estimates as compared with conventional methods.

A potential fatal problem for electrical power delivery through power lines in Northern countries is when snow or ice accumulates on electrical insulators. This could lead to snow or ice-induced outages and voltage collapse, causing huge economic loss. Further, large swing angles due to wind may cause short circuits. This paper presents a novel video surveillance system for detecting snow coverage on electric insulators and swing angles of insulators using videos from a remote outdoor 420 kV power transmission line. To the best of our knowledge, it is the first insulator snow surveillance system base on automatic image analysis techniques. We propose using hybrid techniques by combining histograms, boundaries and template cross-correlations for analyzing a broad range of scenarios caused by changing weather and lighting conditions. Experiments on videos captured during several month periods have shown promising and valuable estimation results. For image pixels related to snow on insulators, our system has yielded an average detection rate of 93% for good quality images and 67.6% for poor quality images, and a corresponding average false alarm of 9% and 18.1%.

We propose a novel scheme that jointly employs anisotropic mean shift and particle filters for tracking moving objects from video. The proposed anisotropic mean shift, that is applied to partitioned areas in a candidate object bounding box whose parameters (center, width, height and orientation) are adjusted during the mean shift iterations, seeks multiple local modes in spatial-kernel weighted color histograms. By using a Gaussian distributed Bhattacharyya distance as the likelihood and mean shift updated parameters as the state vector, particle filters become more efficient in terms of tracking using a small number of particles (<20). The combined scheme is able to maintain the merits of both methods. Experiments conducted on videos containing deformable objects with long-term partial occlusions and intersections have shown robust tracking performance. Comparisons with two existing methods have been made which showed marked improvement in terms of robustness to occlusions, tightness and accuracy of tracked box, and tracking drift.

We propose a novel tracking scheme that jointly employs point feature correspondences and object appearance similarity. For selecting point correspondences, we use a subset of scaleinvariant point features from SIFT that agree with a pre-defined affine transformation. The selected consensus points are then used for pre-selecting candidate regions. For appearance similarity based tracking, we employ an existing anisotropic mean shift, from which the formula for estimating bounding box parameters (width, height, orientation and center) are derived. A switching criterion is utilized to handle the situation where only a small number of point correspondences is found. Experiments and evaluation are performed on tracking moving objects on videos where objects may contain partial occlusions, intersection, deformation and pose changes among other transforms. Our comparisons with two existing methods have shown that the proposed scheme has yielded marked improvement, especially in terms of reducing tracking drifts, of robustness to occlusions, and of tightness and accuracy of tracked bounding box.

This paper addresses issues on moving object tracking from videos. We propose a novel tracking scheme that jointly exploits local object features using consensus point correspondences, and global object appearance and shape models using adaptive particle filter-based eigen-tracking. The paper include the following main novelties: (a) employ consensus feature point correspondences to estimate the motion vector of shape model; (b) employ adaptive particle filters and motion corrected state vector for joint appearance- and shape-based eigen-tracking. An adaptive number of particles is chosen automatically based on an updated estimation of covariance matrix. Further, online learning is made adaptive to avoid learning using partially-occluded objects. The proposed scheme is realized by integrating SURF and RANSAC for estimating consensus point correspondences, and modify an existing particle filter-based eigen-tracking . Experiment results on tracking moving objects in videos have shown that the proposed scheme provides more accurate tracking, especially for objects with fast motion or long-term partial occlusions. The average number of particles is significantly reduced. Comparisons have been made with an existing method, results have shown that the proposed scheme has provided an improved tracking accuracy at the cost of more computations.

This paper proposes a robust object tracking method in video where the time-varying principal components of object's appearance are updated online. Instead of directly updating the PCA-based subspace using matrix decomposition, the sub-space is updated by tracking on the Grassmann manifold. The object tracker performs two alternating processes: (a) online learning of principal component subspace; (b) tracking a moving object using the learned subspace and a particle filter. Learning a PCA-based subspace is performed by treating principal component decompositions as noisy measurements. The measurements are mapped onto the Lie algebra of the Grassmann manifold. The direction of movement of the subspace is then tracked in the Lie algebra using a Kalman filter. The filtered output is then mapped back onto the Grassmann surface to update the principal component-based subspace. This produces a more reliable learning of the subspace. Experiments have been conducted on face image sequences where heads were tilted in variable speed, partial face occlusion, along with changes in object depth and in illuminations. The results and evaluations have shown that the proposed method is robust against these changes when tracking moving objects.

This paper deals with tracking of face blobs containing pose changes. We propose a novel tracking method to deal with face pose changes during the tracking. In the method, tracking is formulated as an approximate solution to the MAP estimate of the state vector, consisting of a linear and a nonlinear part. Multi-pose face appearances are described by local linear models, each being related to a single pose and estimated by probabilistic PCA (PPCA). A Markov model with pose indices as its states is used to model the transitions between poses. Shape and locations of face blobs and the associated pose indices are assumed to be nonlinear, and are estimated by a Rao-Blackwellized particle filter (RBPF). This enables a separate estimation of the linear state vector through marginalizing the joint probability. The proposed method has been tested for videos containing frequent face pose changes and large illumination variations, where 5 poses (left, frontal, right, up, down) were modeled. The tracking results are shown to be robust to variable speed of pose changes and with relatively tight boxes.

Efficiency and robustness are the two most important issues for multiobject tracking algorithms in real-time intelligent video surveillance systems. We propose a novel 2.5-D approach to real-time multiobject tracking in crowds, which is formulated as a maximum a posteriori estimation problem and is approximated through an assignment step and a location step. Observing that the occluding object is usually less affected by the occluded objects, sequential solutions for the assignment and the location are derived. A novel dominant color histogram (DCH) is proposed as an efficient object model. The DCH can be regarded as a generalized color histogram, where dominant colors are selected based on a given distance measure. Comparing with conventional color histograms, the DCH only requires a few color components (31 on average). Furthermore, our theoretical analysis and evaluation on real data have shown that DCHs are robust to illumination changes. Using the DCH, efficient implementations of sequential solutions for the assignment and location steps are proposed. The assignment step includes the estimation of the depth order for the objects in a dispersing group, one-by-one assignment, and feature exclusion from the group representation. The location step includes the depth-order estimation for the objects in a new group, the two-phase mean-shift location, and the exclusion of tracked objects from the new position in the group. Multiobject tracking results and evaluation from public data sets are presented. Experiments on image sequences captured from crowded public environments have shown good tracking results, where about 90% of the objects have been successfully tracked with the correct identification numbers by the proposed method. Our results and evaluation have indicated that the method is efficient and robust for tracking multiple objects (>or= 3) in complex occlusion for real-world surveillance scenarios.

Video surveillance has drawn increasing interests in recent years. This paper addresses the issue of moving object tracking from videos. A two-step processing procedure is proposed: an incremental 2DPCA (two-dimensional principal component analysis)-based method for characterizing objects given the tracked regions, and a ML (maximum likelihood) blob-tracking process given the object characterization and the previous blob sequence. The proposed incremental 2DPCA updates the row- and column-projected covariance matrices recursively, and is computationally more efficient for online learning of dynamic objects. The proposed ML blob-tracking takes into account both the shape information and object characteristics. Tests and evaluations were performed on indoor and outdoor image sequences containing a range of single moving object in dynamic backgrounds, which have shown good tracking results. Comparisons with the method using the conventional PCA were also made.

This paper addresses the issues of nonlinear edge-preserving image smoothing and segmentation. A ML-based approach is proposed which uses an iterative algorithm to solve the problem. First, assumptions about segments are made by describing the joint probability distribution of pixel positions and colours within segments. Based on these assumptions, an optimal smoothing algorithm is derived under the ML condition. By studying the derived algorithm, we show that the solution is related to a two-stage mean shift which is separated in space and range. This novel ML-based approach takes a new kernel function. Experiments have been conducted on a range of images to smooth and segment them. Visual results and evaluations with 2 objective criteria have shown that the proposed method has led to improved results which suffer from less over-segmentation than the standard mean-shift.

Video surveillance and object tracking have drawn increased interests in recent years. This paper addresses the problem of moving object tracking from image sequences captured from stationary cameras. Based on the previous work on video segmentation using joint space-time-range mean shift, we extend the scheme to enable the tracking of moving objects. Large displacements of pdf modes in consecutive image frames are exploited for tracking. We also improve the above mean shift-based video segmentation by introducing edge-guided merging of over-segmented regions. This can be viewed as an extension of the enhanced mean shift 2D image segmentation in [2] to the enhanced space-time-range mean shift video segmentation. Experiments have been conducted on several indoor and outdoor videos. Our preliminary results and performance evaluation have indicated the effectiveness of the proposed scheme.

A new 3D face representation and recognition approach is presented in this paper. Two sets of facial curves are extracted from a face range image, and a novel facial feature representation, the affine integral invariant, is introduced to mitigate the effect of pose on the facial curves. A human face is shown to be representable by a small subset of those affine integral invariant curves. A recognition procedure based on the discriminant analysis and Jensen-Shannon divergence analysis is proposed. Substantiating examples are provided with an achieved classification accuracy of 92.57%

This paper proposes a novel region-based scheme for dynamically modeling time-evolving statistics of video background, leading to an effective segmentation of foreground moving objects for a video surveillance system. In (L. Li et al., 2004) statistical-based video surveillance systems employ a Bayes decision rule for classifying foreground and background changes in individual pixels. Although principal feature representations significantly reduce the size of tables of statistics, pixel-wise maintenance remains a challenge due to the computations and memory requirement. The proposed region-based scheme, which is an extension of the above method, replaces pixel-based statistics by region-based statistics through introducing dynamic background region (or pixel) merging and splitting. Simulations have been performed to several outdoor and indoor image sequences, and results have shown a significant reduction of memory requirements for tables of statistics while maintaining relatively good quality in foreground segmented video objects.

This paper proposes a music signal synthesis scheme that is based on sinusoid modeling and sliding-window ESPRIT. Despite widely used audio coding standards, effectively synthesizing music using sinusoid models, more suitable for harmonic rich music signals, remains an open issue. In the proposed scheme, music signals are modeled by a sum of damped sinusoids in noise. A sliding window ESPRIT algorithm is applied. A continuity constraint is then imposed for tracking the time trajectories of sinusoids in music and for removing spurious spectral peaks in order to adapt to the changing number of sinusoid contents in dynamic music. Simulations have been performed to several music signals with a range of complexities, including music recorded from banjo, flute and music with mixed instruments. The results from listening and spectrograms have strongly indicated that the proposed method is very robust for music synthesis with good quality

This paper proposes a novel error-resilient packet video codec through the use of harmonic frame-expansions. It provides robustness to packet video in erasure channels in an integrated framework without requiring inter/intra coding modes. In the proposed method, spatial redundancy is added by applying a frame-expansion to DCT coefficients while temporal redundancy is added by filtering the motion compensated reference blocks. The frame expansion provides excellent error-resilience at low frequencies, while filtering of motion-reference blocks prevents the propagation of errors at high frequencies. Mathematical expressions of the PSNR for packet video streams with arbitrary losses are derived using piecewise linear approximations. Preliminary simulation results have shown that the proposed codec has generated high quality video at low bit-rates with significant packet losses, and has much less visual artifact than the conventional codec's with similar PSNRs.

This paper proposes a novel error-resilient packet video coding method which is designed to operate on unreliable IP networks suffering from large bursts of packet losses. The main novelties of the proposed system are: (1) A Bayesian packet loss predictor is employed to dynamically predict the packet loss probabilities; (2) A harmonic frame-expansion is applied to the video stream to add error-resilience. This includes dynamically choosing the size of the expansions. (3) Error-propagation is reduced by filtering out error-prone high-frequency components from reference images. This means that the proposed video codec method does not need to switch between the inter and intra coding modes. (4) Theoretical analysis of the video performance is given. Our preliminary simulations have shown that the frame expansion method can provide excellent resilience to video packet losses. The system has generated high quality videos at low bit-rates (64kbs) on simulated networks with the packet loss probability ranging from 5% to 20%. Visual inspection has shown that under similar PSNRs, the resulting videos have noticeable less visual artifacts as compared with those from the conventional codecs

This paper addresses the problem of background modeling for foreground object detection in complex environments. A Bayesian framework that incorporates spectral, spatial, and temporal features to characterize the background appearance is proposed. Under this framework, the background is represented by the most significant and frequent features, i.e., the principal features, at each pixel. A Bayes decision rule is derived for background and foreground classification based on the statistics of principal features. Principal feature representation for both the static and dynamic background pixels is investigated. A novel learning method is proposed to adapt to both gradual and sudden "once-off" background changes. The convergence of the learning process is analyzed and a formula to select a proper learning rate is derived. Under the proposed framework, a novel algorithm for detecting foreground objects from complex environments is then established. It consists of change detection, change classification, foreground segmentation, and background maintenance. Experiments were conducted on image sequences containing targets of interest in a variety of environments, e.g., offices, public buildings, subway stations, campuses, parking lots, airports, and sidewalks. Good results of foreground detection were obtained. Quantitative evaluation and comparison with the existing method show that the proposed method provides much improved results.

This paper proposes a novel method for detection and segmentation of foreground objects from a video which contains both stationary and moving background objects and undergoes both gradual and sudden "once-off" changes. A Bayes decision rule for classification of background and foreground from selected feature vectors is formulated. Under this rule, different types of background objects will be classified from foreground objects by choosing a proper feature vector. The stationary background object is described by the color feature, and the moving background object is represented by the color co-occurrence feature. Foreground objects are extracted by fusing the classification results from both stationary and moving pixels. Learning strategies for the gradual and sudden "once-off" background changes are proposed to adapt to various changes in background through the video. The convergence of the learning process is proved and a formula to select a proper learning rate is also derived. Experiments have shown promising results in extracting foreground objects from many complex backgrounds including wavering tree branches, flickering screens and water surfaces, moving escalators, opening and closing doors, switching lights and shadows of moving objects.

To better understand the environment we are living, especially animals and birds around us, we need to study their behavior and the way of their communication. This paper addresses the problem of classifying bird species of interests using the digital signals of recorded bird songs. First, through comparisons of speech and bird song signals and the experiments, we propose a simple model (similar to that of the speech) for generating synthetic bird songs. We then propose a key-bird-song searching method for the recognition of bird species of interest. This is possible since bird songs appear to be simpler as compared to human speech. A hierarchical classification method is then suggested. In the coarse level of the classification, only candidate songs from those birds whose time-dependent coupled sound patterns are 'close' to that of the species of interest are chosen as the candidates. In the fine level, time-frequency 'format' trajectory-related features from the candidate songs are used for the classification. A case study is conducted for the recognition of a selected bird species, the Great Tit. Preliminary experimental results from using five different bird species and 87 songs have shown promising results in recognizing the selected bird species of interest, with less than 3% of classification errors.

This paper proposes a novel method for tracking persons based on the principal colors of human objects. First, an efficient human object representation method, principal color representation (PCR), is proposed. Asymmetric similarity measures are then proposed based on the principal color representation. These asymmetric similarity measures could be used to evaluate the matching between two individuals as well as visual evident of an individual in a group. An efficient algorithm for tracking persons as individuals or in groups is then described. The method has been tested using image sequences containing multiple moving persons frequently gathering and separating. Our test results have shown that proposed method has successfully tracked both persons as individuals or in groups, and is robust to illumination changes.

Power system transients cause problems and disturbances that might be harmful to electronic equipment. However, very little has been done to systematically understand and analyze power system transients. There is also a lack of computational effective methods as transients are short and associated with a wide range of events and system conditions. This paper addresses the following issues: (1) to categorize the power system transients according to their underlying causes; (2) to propose a signal processing method, ESPRIT, for the analysis of a range of transients due to different causes. Experimental results are included to evaluate the usefulness of ESPRIT in power system transient analysis. Remaining open issues for research are also discussed.

This paper proposes a novel method for detecting foreground objects in nonstationary complex environments containing moving background objects. We derive a Bayes decision rule for classification of background and foreground changes based on inter-frame color co-occurrence statistics. An approach to store and fast retrieve color co-occurrence statistics is also established In the proposed method, foreground objects are detected in two steps. First, both foreground and background changes are extracted using background subtraction and temporal differencing. The frequent background changes are then recognized using the Bayes decision rule based on the learned color co-occurrence statistics. Both short-term and longterm strategies to learn the frequent background changes are proposed Experiments have shown promising results in detecting foreground objects from video containing wavering tree branches and flickering screens/water surface. The proposed method has shown better performance as compared with two existing methods.

Presents a novel method which uses a special type of multi-scale isotropic band-pass filters to detect image landmark and corner features. This paper mainly contributes to the solutions of the following problems: (1) defining a general family of feature detectors under a unified framework that are able to enhance and detect the desired features; (2) explaining theoretically and experimentally why these feature points can be detected by the proposed detectors; and (3) extending the detectors to multi-scale versions for jointly achieving good detectability and localization where an automatic scale selection method is applied. The paper then presents several applications for detecting landmarks and corners using the proposed methods, in order to illustrate their usage. These include detecting landmarks from gesture images (of the face and hand), from airborne and vehicle-borne IR landmine images, and from images containing object corners. Experiments have been performed using the proposed detectors to these applications. Some comparisons and evaluations have also been performed. The results have demonstrated the effectiveness of the proposed detectors in terms of feature detectability and localization.

A novel adaptive nonlinear filter is proposed aimed at smoothing homogenous regions while maintaining image structures. The filter can be utilized as a pre-processing tool in image segmentation and edge estimation for improving the results. Several special features are introduced to the filter, including using local adaptive radial clustering and pixel filtering to exclude the influence of outliers and to maintain image structures; using the steepest-ascent method to iteratively update pixels to the nearest clusters obtained by mean-shift; and introducing highly parallel processing by using random seed samples and their associated data blocks which enables fast processing and the global optimum solution of the nonlinear filter. Experiments were done on images of various complexities, and good results were obtained. Evaluations of the filter were also done in terms of edge preservation and image segmentation.

A novel adaptive mean shift filter is proposed for unsupervised color image segmentation. Segmentation is obtained by iteratively estimating local clusters and modifying (filtering) pixels along the steepest ascent towards their nearest clusters using local data from randomly partitioned image, followed by a simple post processing. Several new steps are introduced to avoid pixels drifting towards shaded regions, pixel filtering using outliers and other local mode, and the blur of image structures. In each local area, the number of clusters is set to be adaptive according to the dynamic range of the local histogram. The filter is applied only to a radial set of pixels chosen from the color histogram, instead of all pixels in the local area. Further, the random partitions of the image make it possible for parallel processing of each local area, hence the potential of fast processing. Experiments were performed on color images with different complexity, and good segmentation results were obtained. Some preliminary evaluations were also performed using the uniformity measure.

This paper addresses the problems of object corner detection and the applications of corners to affine-similar object matching for object retrieval from a database. Two simple and efficient corner detection methods are presented. One is based on detecting sharp angles on a smoothed object boundary curve, and the other is based on a 2D rotationally symmetric bandpass filter applied onto an image. Two object matching approaches which exploit these corners are then investigated. In the first approach, affine objects are retrieved by matching their features derived from the corners. Dissimilarity measures for objects are formulated based on these features. In the second approach, affine objects are retrieved by matching object boundary curves modeled by a constrained active B-spline curve model. The model differs from the conventional B-spline by retaining significant object corners as a subset of B-spline knot points. This enables better correspondences of corners and easy selection of starting points for modeled curves, and hence a more accurate affine object matching. Various images and object boundary curves were used to verify the proposed methods, and the experimental results are convincing.

In this paper, we consider the problem of matching 2D planar object curves from a database, and tracking moving object curves through an image sequence. The first part of the paper describes a curve data compression method using B-spline curve approximation. We present a new constrained active B-spline curve model based on the minimum mean square error (MMSE) criterion, and an iterative algorithm for selecting the "best" segment border points for each B-spline curve. The second part of the paper describes a method for simultaneous object tracking and affine parameter estimation using the approximate curves and profiles. We propose a novel B-spline point assignment algorithm which incorporates the significant corners for interpolating corresponding points on the two curves to be compared. A gradient-based algorithm is presented for simultaneously tracking object curves, and estimating the associated translation, rotation and scaling parameters. The performance of each proposed method is evaluated using still images and image sequences containing simple objects.

One problem in tracking an object through video frames and estimating the associated motion parameters is that the object of interest may undergo various affine transformations and a small deformation. This paper presents a multi-stage algorithm for tracking moving objects using joint segmentation and B-spline contour modelling and motion parameters estimation. The segmentation algorithm uses merging thresholds which are based on the local image property. A fast gradient-based B-spline contour matching algorithm and a robust knot re-assignment method are proposed for tracking moving objects through image frames and estimating their motion parameters. The application of the scheme on images of hand and tool movements, and an image sequence containing a screwdriver with affine transformations shows that the estimated parameters are in very close agreement with the actual parameters.

The recursive-least-squares (RLS) sliding-window lattice is extended and completed by including the functions of real-time window-length decrement and increment based on vector-space geometric projections. Such capabilities improve the tracking performance of the filter for time-varying signals with different rates of variation by eliminating poor data and incorporating good data in the memory. An RLS circular lattice with real-time variation of the sliding-window length is developed for scalar processing of multivariate time-varying signals. Computer simulation results have verified that better filter tracking capability has been obtained in adaptive prediction of one- and two-dimensional time-varying signals.>