Dependency relations among visual entities are ubiquity because both objects and scenes are highly structured. They provide prior knowledge about the real world that can help improve the generalization ability of deep learning approaches. Different from contextual reasoning which focuses on feature aggregation in the spatial domain, visual dependency reasoning explicitly models the dependency relations among visual entities. In this paper, we introduce a novel network architecture, termed the dependency network or DependencyNet, for semantic segmentation. It unifies dependency reasoning at three semantic levels. Intra-class reasoning decouples the representations of different object categories and updates them separately based on the internal object structures. Inter-class reasoning then performs spatial and semantic reasoning based on the dependency relations among different object categories. We will have an in-depth investigation on how to discover the dependency graph from the training annotations. Global dependency reasoning further refines the representations of each object category based on the global scene information. Extensive ablative studies with a controlled model size and the same network depth show that each individual dependency reasoning component benefits semantic segmentation and they together significantly improve the base network. Experimental results on two benchmark datasets show the DependencyNet achieves comparable performance to the recent states of the art.

In recent years, video surveillance has become essential for security applications used to monitor many organizations and locations, and it is therefore important to ensure the reliability of these surveillance videos. Unfortunately, surveillance videos can be forged with little effort by deleting an object from a video scene while leaving no visible traces. A fundamental challenge in video security is to determine whether or not an object has been removed from a video. This task is particularly challenging due to the lack of ground truth bases that can be used to verify the originality and integrity of video contents. In this paper, we propose a novel approach based on sequential and patch analyses to detect object removal forgery and to localize forged regions in videos. Sequential analysis is performed by modeling video sequences as stochastic processes, where changes in the parameters of these processes are used to detect a video forgery. Patch analysis is performed by modeling video sequences as a mixture model of normal and anomalous patches, with the aim to separate these patches by identifying the distribution of each patch. We localize forged regions by visualizing the movement of removed objects using anomalous patches. We conduct our experiments at both pixel and video levels to determine the effectiveness and efficiency of our approach to detection of video forgery. The experimental results show that our approach achieves excellent detection performance with low-computational complexity and leads to robust results for compressed and low-resolution videos.

The lack of interpretability in current deep learning models causes serious concerns as they are extensively used for various life-critical applications. Hence, it is of paramount importance to develop interpretable deep learning models. In this paper, we consider the problem of blind deconvolution and propose a novel model-aware deep architecture that allows for the recovery of both the blur kernel and the sharp image from the blurred image. In particular, we propose the Deep Unfolded Richardson-Lucy (Deep-URL) framework - an interpretable deep-learning architecture that can be seen as an amalgamation of classical estimation technique and deep neural network, and consequently leads to improved performance. Our numerical investigations demonstrate significant improvement compared to state-of-the-art algorithms.

Sparse coding is widely used in image denoising, deblurring, clustering, and classification. However, most existing approaches to sparse coding failed to consider the fact that high dimensional data naturally reside on geometrical structure of the data space. It has been shown that geometric information of the data is important for both inversion and discrimination. In this paper, we proposed a generalized framework for image restoration and representation by combining sparse coding and graph based algorithms. In image denoising and deblurring problems, an image is first decomposed into cartoon layer (piecewise-smooth contents) and texture layer (textures and sharp edges) using morphological component analysis (MCA); then optimal graph Laplacian regularizer (OGLR) algorithm and simultaneous sparse coding with Gaussian scale mixture prior (SSC-GSM) algorithm are applied to cartoon layer and texture layer, respectively; final restored image is generated by aggregating the outcomes from two algorithms. The proposed hybrid image restoration algorithm outperforms state-of-the-art image denoising algorithms, such as BM3D on natural images, measured in PSNR, and performs comparatively in image deblurring. In image clustering and classification problems, we convert our generalized framework into a novel dual graph regularized sparse coding method to transform the nonlinear data space and feature space into linear space, two efficient optimization algorithms are provided for the numerical implementation. The experimental results show that our generalized graph Laplacian and sparse coding framework performs competitively with popular denoising, deblurring, clustering, and classification methods.

In digital image steganography, the statistical model of an image is essential for hiding data in less detectable regions and achieving better security. This has been addressed in the literature where different cost-based and statistical model-based approaches were proposed. However, due to the usage of heuristically defined distortions and non-constrained message models, resulting in numerically solvable equations, there is no closed-form expression for security as a function of payload. The closed-form expression is crucial for a better insight into image steganography problem and also improving performance of batch steganography algorithms. Here, we develop a statistical framework for image steganography in which the cover and the stego messages are modeled as multivariate Gaussian random variables. We propose a novel Gaussian embedding model by maximizing the detection error of the most common optimal detectors within the adopted statistical model. Furthermore, we extend the formulation to cost-based steganography, resulting in a universal embedding scheme that improves empirical results of current cost-based and statistical model-based approaches. This methodology and its presented solution, by reason of assuming a continuous hidden message, remains the same for any embedding scenario. Afterward, the closed-form detection error is derived within the adopted model for image steganography and it is extended to batch steganography. Thus, we introduce Adaptive Batch size Image Merging steganographer, AdaBIM, and mathematically prove it outperforms the state-of-the-art batch steganography method and further verify its superiority by experiments.

In this paper, we develop a statistical framework for image steganography in which the cover and stego messages are modeled as multivariate Gaussian random variables. By minimizing the detection error of an optimal detector within the generalized adopted statistical model, we propose a novel Gaussian embedding method. Furthermore, we extend the formulation to cost-based steganography, resulting in a universal embedding scheme that works with embedding costs as well as variance estimators. Experimental results show that the proposed approach avoids embedding in smooth regions and significantly improves the security of the state-of-the-art methods, such as HILL, MiPOD, and S-UNIWARD.

In general, the viewing experience of stereoscopic panoramas heavily relies on the accuracy of perceived depth information. However, most of existed stereoscopic panorama stitching methods always focus on the improvement, refinement, and correction during the image alignment or panorama stitching. The impairment to the depth information introduced by later steps is rarely under consideration. Thus, we propose one general post-stitching depth correction technique based on sparsely detected control points. The proposed depth correction can be divided into two steps. The global one aims at minimizing the average depth error by adjusting the relative poses of two panoramas. Then, the local step tries to relieve those regions-based depth issues with the morphing technique. For those regions that still suffer from visible depth error after the global correction, we select control points in those depth anomaly patches and utilize the Thin-Plate-Spline technique to warp those features into their target position. To demonstrate the robustness of the proposed depth adjustment technique, various experiments are conducted under different camera arrangements and scenarios. The result indicates that the stereoscopic panoramas after proposed depth adjustment could deliver more reasonable depth information and bring a better viewing experience.

Deep neural networks (DNNs) have achieved state-of-the-art results in various pattern recognition tasks. However, they perform poorly on out-of-distribution adversarial examples i.e. inputs that are specifically crafted by an adversary to cause DNNs to misbehave, questioning the security and reliability of applications. In this paper, we hypothesize inter-class and intra-class feature variances to be one of the reasons behind the existence of adversarial examples. Additionally, learning low intra-class and high inter-class feature variance help classifiers learn decision boundaries that are more compact and leave less inter-class low-probability "pockets" in the feature space, i.e. less room for adversarial perturbations. We achieve this by imposing a center loss [1] in addition to the regular softmax cross-entropy loss while training a DNN classifier. Intuitively, the center loss encourages DNNs to simultaneously learn a center for the deep features of each class, and minimize the distances between the intra-class deep features and their corresponding class centers. Our results on state-of-the-art architectures tested on MNIST, CIFAR-10, and CIFAR-100 datasets confirm our hypothesis and highlight the importance of discriminative features in the existence of adversarial examples.

In this paper, we present one saliency-based feature selection and tracking strategy in the feature-based stereoscopic panoramic video generation system. Many existing stereoscopic video composition approaches aim at producing high-quality panoramas from multiple input cameras; however, most of them directly operate image alignment on those originally detected features without any refinement or optimization. The standard global feature extraction threshold always fails to guarantee stitching correctness of all human interested regions. Thus, based on the originally commonly identified feature set, we incorporate the saliency map into the distribution of control points to remove the redundancy in texture-rich regions and ensure the adequacy of selected features in visual sensitive regions. Intuitively, under the guidance of saliency change in the video sequence, one grid-based feature updating strategy is operated between consecutive frames instead of the standard global feature updating. The experiments show that our method can improve the stitching quality of visual important region without impairment to the human less-interested regions in the generated stereoscopic panoramic video.

Recently, sparse coding has been widely and successfully used in image classification, noise reduction, texture synthesis and audio processing. Although traditional sparse coding method with fixed dictionaries like wavelet and curvelet can produce promising results, unsupervised sparse coding has shown its advantage by optimizing the dictionary adaptively. However, existing unsupervised sparse coding failed to consider the high dimensional manifold information within data. Recently, a graph regularized sparse coding method has shown outstanding performance by incorporating graph laplacian manifold information. In this paper, we proposed a sparse coding method called locally linear embedded sparse coding, to consider the local manifold structure as well as learning the sparse representation. We also provided a novel modified online dictionary learning method which iteratively utilizes modified least angle regression and block coordinate descent method to solve the problem. Instead of getting entire coefficient matrix then update dictionary matrix, our method updates coefficient vector and dictionary matrix in each inner iteration. Extensive experimental results have demonstrated the efficiency and accuracy of our method in image clustering.

Sparse coding has been widely and successfully used in image classification, noise reduction, texture synthesis, and audio processing. Although existing sparse coding methods can produce promising results, they failed to consider the high dimensional manifold information within data. In this paper, we propose a dual graph regularized sparse coding method to effectively preserve duality between data points and features for sparse representation. This is achieved by Feature-sign Search with Lagrange Dual (FS-LD) algorithm and Least-Angle Regression with Block Coordinate Descent (LARS-BCD) algorithm. Experimental results in clustering and classification show that the proposed method outperforms other existing methods.

This paper presents a novel approach to increase the performance bounds of image steganography under the criteria of minimizing distortion. The proposed approach utilizes a steganalysis convolutional neural network (CNN) framework to understand an image's model and embed in less detectable regions to preserve the model. In other word, the trained steganalysis CNN is used to calculate derivatives of the statistical model of an image with respect to embedding changes. The experimental results show that the proposed algorithm outperforms previous state-of-the-art methods in a wide range of low relative payloads when compared with HUGO, S-UNIWARD, and HILL by the state-of-the-art steganalysis.

In this paper, we present a solution for the generation of high-resolution stereoscopic panoramas. Many existing monocular approaches can produce high-quality panoramas; however, direct application of these techniques to the stereo case often fail to deliver high-quality stereo vision. Compared to generating a high-quality monocular panorama, generating a high-quality stereo panorama using existing techniques is challenging owing to the inconsistency between the left and right views and difficulties in disparity control. In our proposed disparity control strategy, one commonly identified feature set is utilized to describe the identical feature in two pairs of adjacent input images instead of processing the left and right images independently. In addition, the conventional RANSAC algorithm is extended to the stereo case to further improve the stereo consistency. After the aligning and blending steps, a global panorama disparity correction technique is applied to compensate for the vertical disparity and correct horizontal disparity. The experiments show that our method can enforce left and right input images to be stitched by very similar pose parameters and generate high-quality stereo panoramas that provide a pleasant 3D viewing experience.

Precise friend recommendation is an important problem in social media. Although most social websites provide some kinds of auto friend searching functions, their accuracies are not satisfactory. In this paper, we propose a more precise auto friend recommendation method with two stages. In the first stage, by utilizing the information of the relationship between texts and users, as well as the friendship information between users, we align different social networks and choose some “possible friends.” In the second stage, with the relationship between image features and users, we build a topic model to further refine the recommendation results. Because some traditional methods, such as variational inference and Gibbs sampling, have their limitations in dealing with our problem, we develop a novel method to find out the solution of the topic model based on series expansion. We conduct experiments on the Flickr dataset to show that the proposed algorithm recommends friends more precisely and faster than traditional methods.

This paper considers the end-to-end mean squared error distortion in reconstructing a memoryless proper-complex Gaussian source transmitted over a set of parallel block-fading Gaussian noise channels, where the fading gains are modeled as correlated Rayleigh distributed random variables with different average powers, thus resulting in asymmetric average received signal noise ratios (SNRs). The distortion exponent (i.e., how fast the average distortion decays to zero as the average received SNR increases) of several coding strategies based on separate source and channel coding is characterized. The definition of distortion exponent commonly used in the literature for SNR-symmetric channels is generalized to the case of SNR-asymmetric channels. It is shown that fading correlation degrades the achievable mean squared error distortion, but does not affect the distortion exponent in the analyzed achievable schemes. The logarithm of the determinant of the fading correlation matrix is found to be a proxy for measuring the performance degradation due to correlation as compared with the case of independent fading. The proposed framework allows one to study any number of correlated parallel channels, contrary to the most of the literature that restricts attention to two channels only, with the same received SNR and with independent fading. In particular, a scheme based on the multiple description coding with more than two descriptions is analyzed; it is shown that determining the distortion exponent in this setting reduces to solving a linear program, which can be done numerically very efficiently. The proposed methodology relies on combining the ideas from linear innovation sequences and properties of determinant of sub-matrices. Interestingly, it is found that SNR-asymmetry is beneficial for multiple description coding when the total average received SNR in decibel is held constant. Even in the SNR-symmetric case, asymmetry in the compression rates is shown to lead to a larger distortion exponent than symmetric rates.

It has been one year since i assumed my current role as Editor-in-Chief (EIC) of the IEEE Transactions on Circuits and Systems for Video Technology (CSVT). It has been an exciting year for me and the CSVT community and I would like to take a moment to recap some of the major milestones for the transactions over the past year.

Presents the introductory editorial for this issue of the publication.

In the past decade, great efforts have been made to extend linear discriminant analysis for higher-order data classification, generally referred to as multilinear discriminant analysis (MDA). Existing examples include general tensor discriminant analysis (GTDA) and discriminant analysis with tensor representation (DATER). Both the two methods attempt to resolve the problem of tensor mode dependency by iterative approximation. GTDA is known to be the first MDA method that converges over iterations. However, its performance relies highly on the tuning of the parameter in the scatter difference criterion. Although DATER usually results in better classification performance, it does not converge, yet the number of iterations executed has a direct impact on DATER's performance. In this paper, we propose a closed-form solution to the scatter difference objective in GTDA, namely, direct GTDA (DGTDA) which also gets rid of parameter tuning. We demonstrate that DGTDA outperforms GTDA in terms of both efficiency and accuracy. In addition, we propose constrained multilinear discriminant analysis (CMDA) that learns the optimal tensor subspace by iteratively maximizing the scatter ratio criterion. We prove both theoretically and experimentally that the value of the scatter ratio criterion in CMDA approaches its extreme value, if it exists, with bounded error, leading to superior and more stable performance in comparison to DATER.

The IEEE Transactions on Circuits and Systems for Video Technology has been the premier venue for publications in the area of video technology, such as video compression, hardware, and systems, since its inception in 1991. Over the past decade, the scope of CSVT has expanded and it is now the premier journal in all areas related to video technology and video systems, including video analysis and processing. In the next few years, the IEEE Transactions on Circuits and Systems for Video Technology will continue to serve as the main venue for publication related to video technology and video systems and further expand its scope. In particular, CSVT will cover all aspects of visual information relating to video or that have the potential to impact future developments in the field of video technology and video systems. An important change in CSVT policy is that we will no longer limit conference publication prior to submission of a paper to CSVT exclusively to ISCAS. Going forward, papers that have been published in any conference or workshop proceedings may be submitted for consideration to CSVT. If you have published preliminary results of your work in any imaging-related conference (e.g., ICIP, VCIP, ICME, CVPR, ICCV, ICPR, ACM Multimedia, ICC, GLOBECOM, and ISCAS) and would like to expose your work in visual information to the premier publication in video technology and video systems, I encourage you to consider submitting your work to the IEEE Transactions on Circuits and Systems for Video Technology.

The Editor-in-Chief (EiC) introduces the new Deputy EiC, Dr. Shipeng Li, and the 25 new Associate Editors of the IEEE Transactions on Circuits and Systems for Video Technology. This is an outstanding group of world-renowned scholars in all areas of video technology and video systems. Dr. Li and the Associate Editors also serve as members of the Editorial Board of the IEEE Transactions on Circuits and Systems for Video Technology. Short professional biographies are presented for all.

Compressive sensing (CS) has triggered an enormous research activity since its first appearance. CS exploits the signal's sparsity or compressibility in a particular domain and integrates data compression and acquisition, thus allowing exact reconstruction through relatively few nonadaptive linear measurements. While conventional CS theory relies on data representation in the form of vectors, many data types in various applications, such as color imaging, video sequences, and multisensor networks, are intrinsically represented by higher order tensors. Application of CS to higher order data representation is typically performed by conversion of the data to very long vectors that must be measured using very large sampling matrices, thus imposing a huge computational and memory burden. In this paper, we propose generalized tensor compressive sensing (GTCS)-a unified framework for CS of higher order tensors, which preserves the intrinsic structure of tensor data with reduced computational complexity at reconstruction. GTCS offers an efficient means for representation of multidimensional data by providing simultaneous acquisition and compression from all tensor modes. In addition, we propound two reconstruction procedures, a serial method and a parallelizable method. We then compare the performance of the proposed method with Kronecker compressive sensing (KCS) and multiway compressive sensing (MWCS). We demonstrate experimentally that GTCS outperforms KCS and MWCS in terms of both reconstruction accuracy (within a range of compression ratios) and processing speed. The major disadvantage of our methods (and of MWCS as well) is that the compression ratios may be worse than that offered by KCS.

Game-theoretic methods based on Nash equilibria have been widely used in various fields including signal processing and communication applications such as cognitive radio systems, sensor networks, defense networks and gene regulatory networks. Solving the Nash equilibria, however, has been proven to be a difficult problem, in general. It is therefore desired to obtain efficient algorithms for solving the Nash equilibria in various special cases. In this paper, we propose a Compressed-Sensing Game Theory (CSGT) framework to solve the Nash equilibria. We demonstrate that the proposed CSGT framework provides a polynomial complexity solution to the Nash Equilibria, thus allowing more general pay-off functions for certain classes of two-player dynamic games. We also provide numerical examples that demonstrate the efficiency of proposed CSGT framework in solving the Nash equilibria for two-player games in comparison to existing algorithms.

This paper derives a sparse optimal perturbation of gene regulatory networks by determining the optimal perturbation of the minimal number of individual genes that force the network to settle into desired equilibrium states. Previous efforts have led to intervention in gene regulatory networks by deriving the optimal perturbation of the state probability transition matrix. Current technology in molecular biology, however, is limited to perturbation of the state of individual genes, not the state probability transition matrix. Our computer simulation experiments on the Human melanoma gene regulatory network demonstrate the superiority of the proposed approach to gene regulation in comparison to the previous methods based on the marginal of the optimal perturbation of the probability transition matrix of the network.

Compressive sensing (CS) has triggered enormous research activity since its first appearance. CS exploits the signal's sparseness or compressibility in a particular domain and integrates data compression and acquisition. While conventional CS theory relies on data representation in the form of vectors, many data types in various applications such as color imaging, video sequences, and multi-sensor networks, are intrinsically represented by higher-order tensors. Application of CS to higher-order data representation is typically performed by conversion of the data to very long vectors that must be measured using very large sampling matrices, thus imposing a huge computational and memory burden. In this paper, we propose Generalized Tensor Compressive Sensing (GTCS)- a unified framework for compressive sensing of higher-order tensors. GTCS offers an efficient means for representation of multidimensional data by providing simultaneous acquisition and compression from all tensor modes. In addition, we compare the performance of the proposed method with Kronecker compressive sensing (KCS). We demonstrate experimentally that GTCS outperforms KCS in terms of both accuracy and speed.

Higher-order singular value decomposition (HOSVD), a natural multilinear extension of the matrix SVD, computes the orthonormal spaces associated with different modes of the tensor. It is widely employed for feature extraction, dimensionality reduction etc. However, due to the vast quantities of tensor entries involved in calculation, it inevitably suffers from high computational cost, especially when recalculation of HOSVD is frequently required. To address the problem, we prove theoretically that the set of HOSVD unitary matrices of a sub-tensor is equivalent to the corresponding subset of HOSVD unitary matrices of the original tensor. Therefore, if we first arrange all tensors in the database compactly as a higher-order tensor, then we only need to conduct HOSVD once on the total tensor. We subsequently propose a robust HOSVD-based multilinear approach for efficiently indexing and retrieving multifactor data, in responding to various query structures. We also apply the proposed method for indexing and retrieval of multi-camera multi-object motion trajectory. Simulation results demonstrate the superior performance of the proposed approach in terms of both robustness and efficiency.

In this paper, we propose a novel statistical estimation algorithm to stochastic context-sensitive grammars (SCSGs). First, we show that a SCSG model can be solved by decomposing it into several causal stochastic context-free grammars (SCFGs) models and each of these SCFGs models can be solved simultaneously using a fully synchronous distributed computing framework. An alternate updating scheme based approximate solution to multiple SCFGs is also provided under the assumption of a realistic sequential computing framework. A series of statistical algorithms are expected to learn SCFGs subsequently. The SCSGs can be then used to represent multiple-trajectory. Experimental results demonstrate the improved performance of our method compared with existing methods for multiple-trajectory classification.

A general framework for sequential particle filtering on graphs is presented in this paper. We present two new articulated motion analysis and object tracking approaches: the graph-based sequential particle filtering framework for articulated object tracking and its hierarchical counterpart. Specifically, we estimate the interaction density by an efficient decomposed inter-part interaction model. To handle severe self-occlusion, we further formulate high-level inter-unit interaction and develop a hierarchical graph-based sequential particle filtering framework for articulated motion analysis. We rely on the proposed general framework of graph-based particle filtering for articulated motion analysis applications. The resulting experiments further demonstrate the superiority of our approach to tracking compared with existing methods.

In this paper, we propose a novel statistical estimation algorithm to stochastic context-sensitive grammars (SCSGs). First, we show that the SCSGs model can be solved by decomposing it into several causal stochastic context-free grammars (SCFGs) models and each of these SCFGs models can be solved simultaneously using a fully synchronous distributed computing framework. An alternate updating scheme based approximate solution to multiple SCFGs is also provided under the assumption of a realistic sequential computing framework. A series of statistical algorithms are expected to learn SCFGs subsequently. The SGSCs can be then used to represent multiple-trajectory. Experimental results demonstrate the improved performance of our method compared with existing methods for multiple-trajectory classification.

This paper presents a novel 2D-TO-3D conversion approach from a monoscopic 2D image sequence. We propose a particle filter framework for recursive recovery of point-wise depth from feature correspondences matched through image sequences. We formulate a novel 2D dynamics model for recursive depth estimation with the combination of camera model, structure model and translation model. The proposed method utilizes edge-detection-assisted scale-invariant features to avoid lack of edge features in scale-invariant features (SIFT). Furthermore, the depths in the depth map are computed and interpolated using 2D Delaunay triangulation. Finally, a stereo-view generation algorithm is presented for multiple users that uses proposed dynamics model and particle filter framework. Experimental results show that our proposed framework yields superior results.

In this paper, we present a comprehensive analysis of self-dual and m-idempotent operators. We refer to an operator as m-idempotent if it converges after m iterations. We focus on an important special case of the general theory of lattice morphology: spatially variant morphology, which captures the geometrical interpretation of spatially variant structuring elements. We demonstrate that every increasing self-dual morphological operator can be viewed as a morphological center. Necessary and sufficient conditions for the idempotence of morphological operators are characterized in terms of their kernel representation. We further extend our results to the representation of the kernel of m-idempotent morphological operators. We then rely on the conditions on the kernel representation derived and establish methods for the construction of m-idempotent and self-dual morphological operators. Finally, we illustrate the importance of the self-duality and m-idempotence properties by an application to speckle noise removal in radar images.

This paper presents a novel approach of multiple target tracking from multiple collaborative cameras. Firstly, particle filtering for conditional density propagation on graphs to address missing frames from one view is introduced. The Markov Properties and Separation Theorem are used to derive an exact solution for estimation on graphs with missing frames. Furthermore, a distributed multiple target tracking solution from multiple cameras is proposed by using collaborative particle filters. With epipolar geometry constraint, camera collaboration message is delivered between different views by particles. Results demonstrate that our system can deal with missing frames in the presence of occlusions.

In this paper, we first present a theorem that HOSVD-based representation of high-order tensor data provides a robust framework that can be used for a unified representation of the HOSVD of all subtensors. We then propose a general algorithm for robust indexing and retrieval of multiple motion trajectories obtained from a multi-camera system. Guided by our theorem, the unitary transformation matrices of a sub tensor can be very well approximated by a subset of unitary matrices corresponding to the same dimensions of the original tensor. As a result, when dealing with flexible query structure consisting of an arbitrary number of objects and cameras, instead of recalculating unitary transformation matrices of the corresponding subtensor, we can just employ a subset of the original unitary matrices. Simulation results are finally used to illustrate the robustness and efficiency of the proposed approach to multiple trajectory indexing and retrieval from multi-camera systems.

Correlated equilibria are a generalization of Nash equilibria that permit agents to act in a correlated manner and can there fore, model learning in games. In this paper we define a special class of correlated equilibria that have hierarchical structure based on the factor graph. Such factor graph-based structural equilibria are more general than Nash equilibria and can model constrained dependencies than general correlated equilibria. We provide the numerical example for using non cooperative stochastic game model on the gene regulatory network under three solution concepts.

MOTIVATION: Analysis and intervention in the dynamics of gene regulatory networks is at the heart of emerging efforts in the development of modern treatment of numerous ailments including cancer. The ultimate goal is to develop methods to intervene in the function of living organisms in order to drive cells away from a malignant state into a benign form. A serious limitation of much of the previous work in cancer network analysis is the use of external control, which requires intervention at each time step, for an indefinite time interval. This is in sharp contrast to the proposed approach, which relies on the solution of an inverse perturbation problem to introduce a one-time intervention in the structure of regulatory networks. This isolated intervention transforms the steady-state distribution of the dynamic system to the desired steady-state distribution. RESULTS: We formulate the optimal intervention problem in gene regulatory networks as a minimal perturbation of the network in order to force it to converge to a desired steady-state distribution of gene regulation. We cast optimal intervention in gene regulation as a convex optimization problem, thus providing a globally optimal solution which can be efficiently computed using standard toolboxes for convex optimization. The criteria adopted for optimality is chosen to minimize potential adverse effects as a consequence of the intervention strategy. We consider a perturbation that minimizes (i) the overall energy of change between the original and controlled networks and (ii) the time needed to reach the desired steady-state distribution of gene regulation. Furthermore, we show that there is an inherent trade-off between minimizing the energy of the perturbation and the convergence rate to the desired distribution. We apply the proposed control to the human melanoma gene regulatory network. AVAILABILITY: The MATLAB code for optimal intervention in gene regulatory networks can be found online: http://syen.ualr.edu/nxbouaynaya/Bioinformatics2010.html.

In this letter, we extend the first-order Markov chain model commonly used in visual tracking and present a novel framework of visual tracking using high-order Monte Carlo Markov chain. By using graphical models to obtain conditional independence properties, we derive a general expression for the posterior density function of anmth-order hidden Markov model. We subsequently use Sequential Importance Sampling (SIS) to estimate the posterior density and obtain the high-order particle filtering algorithm for visual object tracking. Experimental results demonstrate that the performance of our proposed algorithm is superior to traditional first-order particle filtering (i.e., particle filtering derived based on first-order Markov chain).

In this paper, we propose a communication model of evolution and investigate its information-theoretic bounds. The process of evolution is modeled as the retransmission of information over a protein communication channel, where the transmitted message is the organism's proteome encoded in the DNA. We compute the capacity and the rate distortion functions of the protein communication system for the three domains of life: Archaea, Bacteria, and Eukaryotes. The tradeoff between the transmission rate and the distortion in noisy protein communication channels is analyzed. As expected, comparison between the optimal transmission rate and the channel capacity indicates that the biological fidelity does not reach the Shannon optimal distortion. However, the relationship between the channel capacity and rate distortion achieved for different biological domains provides tremendous insight into the dynamics of the evolutionary processes of the three domains of life. We rely on these results to provide a model of genome sequence evolution based on the two major evolutionary driving forces: mutations and unequal crossovers.

In this paper, we develop a novel solution for particle filtering on general graphs. We provide an exact solution for particle filtering on directed cycle-free graphs. The proposed approach relies on a partial-order relation in an antichain decomposition that forms a high-order Markov chain over the partitioned graph. We subsequently derive a closed-form sequential updating scheme for conditional density propagation using particle filtering on directed cycle-free graphs. We also provide an approximate solution for particle filtering on general graphs by splitting graphs with cycles into multiple directed cycle-free subgraphs. We then use the sequential updating scheme by alternating among the directed cycle-free subgraphs to obtain an estimate of the density propagation. We rely on the proposed method for particle filtering on general graphs for two video tracking applications: 1) object tracking using high-order Markov chains; and 2) distributed multiple object tracking based on multi-object graphical interaction models. Experimental results demonstrate the improved performance of the proposed approach to particle filtering on graphs compared with existing methods for video tracking.

View-invariant representation has been shown to be a powerful tool in classification and retrieval of motion events due to camera motions. Traditional null space representation is invariant only for linear transformations and does not yield high accuracy for camera with non-linear motions. In this paper, we propose a novel general framework for non-linear kernel space invariant representation (NKSI), which is invariant to non-linear transformations due to camera motions with standard perspective transformation. We first derive NKSI and then propose an efficient classification and retrieval system relying on NKSI for archiving and searching motion events consisting of motion trajectories. The simulation results demonstrate superior performance of the proposed systems over traditional approaches.

In this paper, we examine the problem of learning from noise-contaminated data in high-dimensional space. A new learning approach based on projections onto multi-dimensional ellipsoids (POME) is introduced, which is applicable to unsupervised clustering, semi-supervised clustering and classification in high-dimensional noisy data. Unlike the traditional learning techniques, where local information is used for data analysis, the proposed POME-based scheme incorporates a priori information of the data distribution. Experimental results in unsupervised clustering demonstrate the superiority of the proposed POME-based scheme to some well-known clustering algorithms, including the k-means and the hierarchical agglomerative clustering. We also illustrate the effectiveness of our proposed POME-based scheme in semi-supervised learning by simulation.

The loopy belief propagation algorithm (LBP) is known to perform extremely well in many practical problems of probability inference and learning on graphical models, even in presence of multiple loops. Although general necessary conditions for convergence of LBP to a unique fixed point solution are still unknown, various techniques have been explored to understand error propagation when LBP fails to converge. In this paper, we rely on the contractive mapping of message errors to present novel distance bounds between multiple fixed point solutions when LBP does not converge. We give examples of networks where our bounds are tighter than existing ones.

The intervention in gene regulatory networks has been modelled as the Markov decision process problem. However, this approach only allows one external control, which is inadequate in many situations such as drug and gene therapies. In this paper, we propose the non-cooperative stochastic game model for intervention in the genetic regulatory networks as the generalization of the Markov decision process approach and formulate the intervention problem into solving the Nash equilibrium. The definition of equilibrium has been proposed and the existences for both infinite and finite horizon cases have been proven. We provide the numerical example for using non-cooperative stochastic game model on the mammalian cell cycle network. We also compare the results under the Nash equilibrium and independent Markov decision process approach.

In this paper, we present a novel bilinear invariant representation for video classification and retrieval. We rely on the kernel space in functional analysis to formulate a general invariants theory. We show that null-space invariants is a special case of the general theory when the transformation is linear. Subsequently, we derive an invariant basis representation for bilinear transformations. We also extend the basis representation to tensor bilinear invariants. We demonstrate that the proposed bilinear invariant basis provides a much more powerful tool than null-space invariants for video classification and retrieval when the different data elements undergo distinct transformations. Simulation results illustrate the superior performance of the proposed bilinear invariant basis representation compared to traditional approaches to invariant video classification and retrieval.

We describe a novel approach for automatically generating streaming architectures from software programs. While existing systems require user-defined stream models, our method automatically identifies producer-consumer streaming relationships and translates them into streaming architectures. Data streams between producer-consumer kernels are represented using a combination of stream descriptors and CFGs, which are categorized into four stream types. A bridge module is generated based on the stream type in the streaming architecture to facilitate data streaming between each producer-consumer pair. Several optimizations are also developed to improve throughput and parallelism. We demonstrate our results on a FPGA based platform. The automatically generated streaming architectures show 1.5-3x speedups over the non-streaming designs by employing spatial and temporal data independence to increase parallelism.

The eight papers in this special section span across theoretical and practical considerations of various aspects of distributed camera networks, including adaptive sensing, distributed processing, efficient communications, and versatile implementations.

A method is introduced to track the object's motion and estimate its pose directly from 2-D image sequences. Scale-invariant feature transform (SIFT) is used to extract corresponding feature points from image sequences. We demonstrate that pose estimation from the corresponding feature points can be formed as a solution to Sylvester's equation. We show that the proposed approach to the solution of Sylvester's equation is equivalent to the classical SVD method for 3D-3D pose estimation. However, whereas classical SVD cannot be used for pose estimation directly from 2-D image sequences, our method based on Sylvester's equation provides a new approach to pose estimation. Smooth video tracking and pose estimation is finally obtained by using the solution to Sylvester's equation within the importance sampling density of the particle filtering framework. Finally, computer simulation experiments conducted over synthetic data and real-world videos demonstrate the effectiveness of our method in both robustness and speed compared with other similar object tracking and pose estimation methods.

In this paper, we present a novel method for virtual focus and object depth estimation from defocused video captured by a moving camera. We use the term virtual focus to refer to a new approach for producing in-focus image sequences by processing blurred videos captured by out-of-focus cameras. Our method relies on the concept of Depth-from-Defocus (DFD) for virtual focus estimation. However, the proposed approach overcomes limitations of DFD by reformulating the problem in a moving-camera scenario. We introduce the interframe image motion model, from which the relationship between the camera motion and blur characteristics can be formed. This relationship subsequently leads to a new method for blur estimation. We finally rely on the blur estimation to develop the proposed technique for object depth estimation and focused video reconstruction. The proposed approach can be utilized to correct out-of-focus video sequences and can potentially replace the expensive apparatus required for auto-focus adjustments currently employed in many camera devices. The performance of the proposed algorithm is demonstrated through error analysis and computer simulated experiments.

Summary form only given. This talk will explore the use of motion information in video surveillance. We will focus primarily on retrieval and activity recognition of motion events in video and sensor databases characterized by multiple interactive motion trajectories. We will present a framework based on tensor decomposition for indexing and retrieval of multiple motion trajectories in video databases. An efficient method for extraction and insertion of partial information in video databases used for multiple motion trajectory representation based on high-order singular value decomposition of tensors will also be provided. We will present a solution to classification and activity recognition of multiple interactive motion trajectories by deriving an extension of hidden Markov models to multiple dimensions. A closed-form solution for the training and classification general forward-backward, expectation-maximization, and Viterbi algorithms for multiple dimensions will be provided for causal systems. An approach to multiple motion trajectory classification and activity recognition based on distributed multidimensional hidden Markov models will be presented for non-causal systems. We will finally introduce a new approach to view-invariance of motion trajectories for unknown and moving cameras based on a null-space matrix representation of motion trajectory information. We will also extend the null-space representation to tensors for view-invariant indexing and retrieval of multiple motion trajectories from unknown and moving cameras. A method for extraction and insertion of partial information into a video database of multiple motion trajectories represented based on null-space tensors is finally presented.

In this paper, we propose a novel general framework for tensor based null space affine invariants, namely, tensor null space invariants (TNSI) with a linear classifier for high order data classification and retrieval. We first derive TNSI, which is perfectly invariant to multidimensional affine transformations due to camera motions for multiple motion trajectories in consecutive motion events. We subsequently propose an efficient classification and retrieval system relying on TNSI for archiving and searching motion events consisting of multiple motion trajectories. The simulation results demonstrate superior performance of the proposed systems.

In this paper, we propose a novel nonparametric modeling technique, namely Space Kernel Analysis (SKA), as a result of the definition of the space kernel. We analyze the uncertainty of SKA and show that SKA is subjected to the bias/variance dilemma. Nevertheless, we demonstrate that, by a proper choice of the space kernel matrix, SKA is able to balance between the robustness and accuracy and hence outperforms other kernel-based learning methods. The cost function of SKA is derived, and it proves that SKA minimizes the Weighted Least Squared cost function whose weight matrix is diagonal and determined by the space kernel matrix. The parallels between SKA and several other nonparametric modeling techniques are examined. Study shows that the traditional Kernel Regression, General Regression Neural Network, Similarity Based Modeling and Radial Basis Function Network are examples of SKA with specified space kernel matrices.

Motion information is regarded as one of the most important cues for developing semantics in video data. Yet it is extremely challenging to build indexing and browsing tools for video data, particularly when it involves interactive motions of multiple objects. The problem is further complicated when the video archives are dynamically updated, and/or queries contains partial information. An efficient solution would require that the feature space used to represent the data be dynamically updated or downdated to allow frequent additions/deletions and matching of queries. Assuming tensor HOSVD as the feature space, in this paper, we propose two novel algorithms, namely, dynamic tensor HOSVD updating algorithm (DTSV D+) and dynamic tensor HOSVD downdating algorithm (DTSV D-), for dynamically updating and downdating existing tensor HOSVD, without recalculating it from the raw data. The proposed algorithms are robustly applied to both full and partial multiple motion trajectories events with varying number of objects, trajectory lengths, and sampling rates. Simulations on real-world multiple motion trajectories data demonstrate the robustness and accuracy of the proposed algorithms.

In this paper, we present a novel methodology for tracking performance evaluation. Considering the continuity of the image sequences in a video, we define a new measurement called tracking difficulty which incorporates the local sequence information among a small image sequence centered at each frame. We subsequently use a reflective model to formulate tracking difficulty. Tracking difficulty curves can not only illustrate at which parts of the video one tracking algorithm performs well or poor, but also provide a way to compare the performance of different tracking algorithms. We further add perturbation analysis to the reflective model to examine how sensitive the tracking algorithm is to noise. Results on data sets are presented to show the effectiveness of our evaluation method.

This paper considers the end-to-end mean-square distortion in reconstructing a memoryless proper-complex Gaussian source transmitted over parallel block-fading Rayleigh AWGN channels. We characterize the distortion exponent of several source and channel coding strategies; that is, we characterize how fast the distortion decays to zero as the SNR increases. Unlike previous works, we consider networks with different received SNR's and with correlated fading. We generalize the definition of distortion exponent to SNR-asymmetric channels. We show that fading correlation degrades the achievable mean-square distortion but does not affect the distortion exponent. The performance degradation is measured in terms of power-offset; that is, the power increment needed to achieve the same performance as the uncorrelated case. We show that the power-offset is proportional to the determinant of the fading correlation matrix. Our proposed methodology allows us to study any number of parallel channels and we are no longer restricted to two channels (as was commonly done in the previous literature). Finally, we show that determining the distortion exponent of multiple description coding (MDC) schemes in high SNR reduces to solving a linear programming problem.

In this paper, we present a general theory of adaptive mathematical morphology (AMM) in the Euclidean space. The proposed theory preserves the notion of a structuring element, which is crucial in the design of geometrical signal and image processing applications. Moreover, we demonstrate the theoretical and practical distinctions between adaptive and spatially-variant mathematical morphology. We provide examples of the use of AMM in various image processing applications, and show the power of the proposed framework in image denoising and segmentation.

In this paper, we introduce a method to estimate the object's pose from multiple video cameras. We derive a centralized solution to pose estimation from multiple video cameras by solving a general matrix equation. Moreover, we provide an equivalent distributed solution to the pose estimation problem based on the independent pose estimation obtained from each camera. We demonstrate that both methods generate superior estimates to the results obtained from any specific camera view. The resulting pose estimation technique is shown to be robust to occlusion and errors from specific camera views, and the computational complexity of the distributed solution grows linearly with the number of cameras. Furthermore, the proposed approach does not require feature matching among images from different camera views nor does it demand reconstruction of 3D points.

In this paper, we present a general framework for analysis of plenoptic sampling by investigating the spectral analysis of plenoptic imaging. The proposed framework provides a unified representation that generalizes several existing methods for plenoptic sampling. The plenoptic sampling theory presented extends the study of plenoptic imaging to parallel cameras that are not restricted to lie on a plane corresponding to a camera array. Furthermore, we conduct an analysis of plenoptic sampling for unstructured camera systems, in which the cameras can reside at arbitrary locations and orientations. Finally, we establish necessary and sufficient conditions for unoccluded imaging, which is a critical precondition for plenoptic sampling. These conditions are shown to introduce constraints on the position and curvature of the surface of the scene as well as the position and orientation of the cameras.

Event/object classification and recognition is an extremely challenging problem, particularly when the query or stored data undergo an affine transformation due to camera motion. The complexity of the problem is further compounded when the input or stored data contain only partial information (e.g. due to object occlusion). Most of the existing representation do not allow view invariant representation and dynamic updating and downdating with in a single framework. In this paper, we present a novel robust multi-dimensional Localized Null Space and associated dynamic updating and downdating techniques, thus allowing classification and retrieval in the presence of affine transformations and partial information. We investigate the robustness of Localized Null Space using perturbation analysis. We further determine the optimal segmentation of the data by minimizing a distortion criterion. We demonstrate the effectiveness and robustness of the proposed techniques for motion event classification and retrieval applications by posing different affine transformations of partial queries.

Motion trajectory is one of the most important cues for extracting semantic information from video data. Numerous studies have focused on the analysis and comparison of the similarity among motion trajectories. Most of the previous methods rely on a global measure that does not account for partial or lost information. In particular, existing techniques fail to capture the salient features of local events shared by distinct motion trajectories. In this paper, we propose a novel local similarity alignment based method for retrieving similar motion events. Our approach is motivated by the well-known global sequence alignment and BLAST algorithms used in character string matching and genomic analysis. The proposed local similarity alignment measure focuses on key segments of motion trajectories and thus yields superior computational speed while providing improved performance, especially in the presence of missing information. We conduct extensive computer simulations that demonstrate the superiority of the proposed approach in terms of its efficiency and computational speed. Moreover, we highlight the robustness of the proposed local similarity alignment method to information loss (e.g. due to occlusion) by demonstrating its performance for partial motion trajectories.

In this paper, we develop novel solutions for particle filtering on graphs. An exact solution of particle filtering for conditional density propagation on directed cycle-free graphs is performed by a sequential updating scheme in a predetermined order. We also provide an approximate solution for particle filtering on general graphs by splitting the graphs with cycles into multiple directed cycle-free subgraphs. We utilize the proposed solution for distributed multiple object tracking. Experimental results show the improved performance of our method compared with existing methods for multiple object tracking.

In this paper, we present new results in performance analysis of super-resolution (SR) image reconstruction. We investigate bounds on the improvement in resolution that can be achieved and its relation to the image sequence. We derive lower bounds on the resolution enhancement factor based on a frequency-domain SR algorithm. We subsequently show that the bounds remain valid for other SR algorithms. Moreover, we consider an image sequence model in the presence of affine motion. We demonstrate theoretically and experimentally that incorporation of affine motion into the image model can be used to increase the enhancement factor in comparison to purely translational motion. Finally, we discuss the extension of the performance bounds to temporal super-resolution methods and its implications on the image sequence.

With the proliferation of reconfigurable systems and flexible memory architectures, there has been intense interest in stream systems. While the existing stream systems require the programs to be written using special models, this paper demonstrates an approach to automatically generate stream programs from existing applications written for non-stream scalar processors. As a part of this approach, we provide a new comparison methodology and an algorithm to automatically generate stream descriptions. A second algorithm identifies processing kernels that can be pipelined. We demonstrate our results on an FPGA based platform.

Particle filters have revolutionized object tracking in video sequences. The conventional particle filter, also called the CONDENSATION filter, uses the state transition distribution as the proposal distribution, from which the particles are drawn at each iteration. However, the transition distribution does not take into account the current observations, and thus many particles can be wasted in low likelihood regions. One of the most popular methods to improve the performance of particle filters relied on the motion-based proposal density. Although the motivation for motion-based particle filters could be explained on an intuitive level, up until now a mathematical rationale for the improved performance of motion-based particle filters has not been presented. In this letter, we investigate the performance of motion-based particle filters and provide an analytical justification of their superiority over the classical CONDENSATION filter. We rely on the characterization of the optimal proposal density, which minimizes the variance of the particles'weights. However, this density does not admit an analytical expression, making direct sampling from this optimal distribution impossible. We use the Kullback-Leibler (KL) divergence as a similarity measure between density functions and denote a particle filter as superior if the KL divergence between its proposal and the optimal proposal function is lower. We subsequently prove that under mild conditions on the estimated motion vector, the motion-based particle filter outperforms the CONDENSATION filter, in terms of the KL performance measure. Simulation results are presented to support the theoretical analysis.

Motion information is regarded as one of the most important cues for developing semantics in video data. Yet it is extremely challenging to build semantics of video clips particularly when it involves interactive motion of multiple objects. Most of the existing research has focused on capturing and modelling the motion of each object individually thus loosing interaction information. Such approaches yield low precision-recall ratios and limited indexing and retrieval performances. This paper presents a novel framework for compact representation of multi-object motion trajectories. Three efficient multi-trajectory indexing and retrieval algorithms based on multilinear algebraic representations are proposed. These include: (i) geometrical multiple-trajectory indexing and retrieval (GMIR), (ii) unfolded multiple-trajectory indexing and retrieval (UMIR), and (iii) concentrated multiple-trajectory indexing and retrieval (CMIR). The proposed tensor-based representations not only remarkably reduce the dimensionality of the indexing space but also enable the realization of fast retrieval systems. The proposed representations and algorithms can be robustly applied to both full and partial (segmented) multiple motion trajectories with varying number of objects, trajectory lengths, and sampling rates. The proposed algorithms have been implemented and evaluated using real video datasets. Simulation results demonstrate that the CMIR algorithm provides superior precision-recall metrics, and smaller query processing time compared to the other approaches.

Video stabilization is an important technique in digital cameras. Its impact increases rapidly with the rising popularity of handheld cameras and cameras mounted on moving platforms (e.g., cars). Stabilization of two images can be viewed as an image registration problem. However, to ensure the visual quality of the whole video, video stabilization has a particular emphasis on the accuracy and robustness over long image sequences. In this paper, we propose a novel technique for video stabilization based on the particle filtering framework. We extend the traditional use of particle filters in object tracking to tracking of the projected affine model of the camera motions. We rely on the inverse of the resulting image transform to obtain a stable video sequence. The correspondence between scale-invariant feature transform points is used to obtain a crude estimate of the projected camera motion. We subsequently postprocess the crude estimate with particle filters to obtain a smooth estimate. It is shown both theoretically and experimentally that particle filtering can reduce the error variance compared to estimation without particle filtering. The superior performance of our algorithm over other methods for video stabilization is demonstrated through computer simulated experiments.

In this paper, we propose an attraction-repulsion expectation-maximization (AREM) algorithm for image reconstruction and sensor field estimation. We rely on a new method for density estimation to address the problems of image reconstruction from limited samples and sensor field estimation from randomly scattered sensors. Density estimation methods often suffer from undesirable phenomena such as over-fitting and over-smoothing. Specifically, various density estimation techniques based on a Gaussian mixture model (GMM) tend to cluster the Gaussian functions together, thus resulting in over-fitting. On the other hand, other approaches repel the Gaussian functions and yield over-smooth density estimates. We propose a method that seeks an equilibrium between over-fitting and over-smoothing in density estimation by incorporating attraction and repulsion forces among the Gaussian functions and determining the optimal balance between the competing forces experimentally. We model the attractive and repulsive forces by introducing the Gibbs and inverse Gibbs distributions, respectively. The maximization of the likelihood function augmented by the Gibbs density mixture is solved under the expectation-maximization (EM) method. Computer simulation results are provided to demonstrate the effectiveness of the proposed AREM algorithm in image reconstruction and sensor field estimation.

In this paper, we propose a novel solution to an arbitrary noncausal, multidimensional hidden Markov model (HMM) for image and video classification. First, we show that the noncausal model can be solved by splitting it into multiple causal HMMs and simultaneously solving each causal HMM using a fully synchronous distributed computing framework, therefore referred to as distributed HMMs. Next we present an approximate solution to the multiple causal HMMs that is based on an alternating updating scheme and assumes a realistic sequential computing framework. The parameters of the distributed causal HMMs are estimated by extending the classical 1-D training and classification algorithms to multiple dimensions. The proposed extension to arbitrary causal, multidimensional HMMs allows state transitions that are dependent on all causal neighbors. We, thus, extend three fundamental algorithms to multidimensional causal systems, i.e., 1) expectation-maximization (EM), 2) general forward-backward (GFB), and 3) Viterbi algorithms. In the simulations, we choose to limit ourselves to a noncausal 2-D model whose noncausality is along a single dimension, in order to significantly reduce the computational complexity. Simulation results demonstrate the superior performance, higher accuracy rate, and applicability of the proposed noncausal HMM framework to image and video classification.

In this paper, we propose a novel robust retrieval and classification system for video and motion events based on null space representation. In order to analyze the robustness of the system, the perturbed null operators have been derived with the first order perturbation theory. Subsequently, the sensitivity of the null operators is discussed in terms of the error ratio and the SNR respectively. Meanwhile, the normwise bounds and componentwise bounds based on classical matrix perturbation theory are presented and discussed. Given the perturbation, uniform sampling are proposed for the convergence of the SNR and Poisson sampling are proposed for the convergence of the error ratio in the mean sense by choosing the rate parameter the same order as the number of samples. The simulation results are provided to demonstrate the effectiveness and robustness of our system in motion event indexing, retrieval and classification that is invariant to affine transformation due to camera motions.

A method is introduced to track the object's motion and estimate its pose from multiple cameras. Firstly pose estimation from one camera is explained. We show that pose estimation from the corresponding feature points can be formed as a solution to Sylvester's equation. Furthermore, we develop a distributed solution, which indicates that pose estimation from multiple cameras can be obtained from the linear combination of results from each single camera. We relay on the results from other cameras to improve the estimate of the first one, and vice versa. Finally, the computer simulation experiments demonstrate the superior performance of our algorithm in robustness.

In this paper, we present two density estimation methods based on constrained expectation-maximization (EM) algorithm. We propose a penalty-based maximum-entropy expectation-maximization (MEEM) algorithm to obtain a smooth estimate of the density function. We further propose an attraction-repulsion expectation-maximization (AREM) algorithm for density estimation in order to determine equilibrium between over-smoothing and over-fitting of the estimated density function. Computer simulation results are used to show the effectiveness of the proposed constrained expectation-maximization algorithms in image reconstruction and sensor field estimation from randomly scattered samples.

In this paper, we propose a novel multi-dimensional distributed hidden Markov model (DHMM) framework. We first extend the theory of 2D hidden Markov models (HMMs) to arbitrary causal multi-dimensional HMMs and provide the classification and training algorithms for this model. The proposed extension of causal multi-dimensional HMMs allows state transitions in arbitrary causal directions and neighbors. We subsequently generalize this framework further to non-causal models by distributing the non-causal models into multiple causal multi-dimensional HMMs. The proposed training and classification process consists of the extension of three fundamental algorithms to multi-dimensional causal systems, i.e. (1) Expectation-Maximization (EM) algorithm; (2) General Forward-Backward (GFB) algorithm; and (3) Viterbi algorithm. Simulation results performed using real-world images and videos demonstrate the superior performance, higher accuracy rate and promising applicability of the proposed DHMM framework.

This paper presents a novel particle allocation approach to particle filtering for articulated object tracking which minimizes the total tracking distortion given a fixed number of particles over a video sequence. Under the framework of decentralized articulated object tracking, we propose the dynamic proposal variance and optimal particle number allocation algorithm for articulated object tracking to allocate particles among different parts of the articulated object as well as different frames. Experimental results show the superior performance of our proposed algorithm to traditional particle allocation methods, i.e. a fixed number of particles for each object part in each frame. To the best of our knowledge, our approach is the first to provide an optimal allocation of a fixed number of particles among different object parts and different frames.

This paper presents a novel classification/retrieval system for motion events based on a perfect view invariant representation of motion trajectories and a linear classifier algorithm. Specifically, Null Space Invariant (NSI) matrix representation for motion trajectories has been derived. The proposed view invariant representation based on the NSI operator is invariant to affine transformations and preserves the null space matrix. We use principal component null space analysis (PC-NSA) for indexing and classification of the NSI operator for recognition and retrieval of motion events. We rely on PC-NSA to determine the distance of the query trajectory to the centroid of each class, which is a statistical information vector in the PCNSA algorithm representing the corresponding motion class. Our results show that NSI provides a powerful approach to motion event recognition and retrieval that is invariant to affine transformations due to camera motion.

The problem of optimal rate allocation to streams constituting multiple description coding (MDC) has largely been addressed under the assumption that the multiple paths available for transmission are uncorrelated. In this paper, we model the transmission paths as correlated erasure channels and then examine the problem of allocating a given coding rate to the multiple descriptions in order to minimize the average distortion of the reconstructed message. We also investigate the relationship between the optimal average distortion and correlation of channels and prove that the bound on the average distortion will decrease as the correlation increases. Furthermore, we derive a closed-form solution for the contour which determines the region where multiple description coding (MDC) or single description coding (SDC) yields the minimal bound on the average distortion. Relying on this contour, we present a heuristic of the optimal rate allocation problem to determine the number of descriptions and relative rates.

In this paper, we study the optimal rate allocation of multiple description coding over multiple erasure channels in order to attain the minimum average distortion of the recovered message at the receiver. The results of this investigation are subsequently used to determine conditions under which the optimal rate allocation is characterized by transmission of one description over a single channel, i.e. single description coding. We formulate the optimal rate allocation problem by using rate- distortion theory under the constraint that the total coding rate is fixed and solve the problem by using numerical methods. Furthermore, we derive a closed-form analytical solution to the optimal rate allocation problem under four conditions and verify that the analytical solution is consistent with the numerical results. Also, we present a heuristic that captures the solution to the optimal rate allocation problem and can be used to determine the number of descriptions and relative rates required to achieve optimality.

This paper presents a novel system for image retrieval and classification based on a robust multi-dimensional view invariant representation and a linear classifier algorithm. Specifically, multi-dimensional null space invariant (NSI) matrix representation has been derived. Moreover, the robustness of null space representation has been investigated with perturbation analysis in terms of the error ratio and SNR. The proposed representation is invariant to affine transformations and preserves the null space matrix. We use principal component null space analysis (PCNSA) of the NSI operator for recognition, indexing, and retrieval of 2D and 3D images. We rely on PCNSA to determine the distance of a query image to the centroid of the class, which is a statistical information vector in the PCNSA algorithm representing the corresponding class of the object. Our results shows that NSI provides a robust and powerful approach to image recognition and classification even when the query image or the stored image has been subjected to unknown affine transformations due to camera motions.

In this paper, we discard the first-order Markov state-space model commonly used in visual tracking and present a framework of visual tracking using high-order Monte Carlo Markov chain. By using graphic models to obtain the conditional independence properties, we derive the general expression of posterior density function for the mth-order hidden Markov model. We subsequently use Sequential Importance Sampling method to estimate the posterior density and obtain the high-order particle filtering algorithm for tracking. Experimental results show the superior performance of our proposed algorithm to traditional first-order particle filtering tracking algorithm, i.e. particle filtering derived based on first-order Markov chain.

BACKGROUND: Over the past decade, many investigators have used sophisticated time series tools for the analysis of genomic sequences. Specifically, the correlation of the nucleotide chain has been studied by examining the properties of the power spectrum. The main limitation of the power spectrum is that it is restricted to stationary time series. However, it has been observed over the past decade that genomic sequences exhibit non-stationary statistical behavior. Standard statistical tests have been used to verify that the genomic sequences are indeed not stationary. More recent analysis of genomic data has relied on time-varying power spectral methods to capture the statistical characteristics of genomic sequences. Techniques such as the evolutionary spectrum and evolutionary periodogram have been successful in extracting the time-varying correlation structure. The main difficulty in using time-varying spectral methods is that they are extremely unstable. Large deviations in the correlation structure results from very minor perturbations in the genomic data and experimental procedure. A fundamental new approach is needed in order to provide a stable platform for the non-stationary statistical analysis of genomic sequences. RESULTS: In this paper, we propose to model non-stationary genomic sequences by a time-dependent autoregressive moving average (TD-ARMA) process. The model is based on a classical ARMA process whose coefficients are allowed to vary with time. A series expansion of the time-varying coefficients is used to form a generalized Yule-Walker-type system of equations. A recursive least-squares algorithm is subsequently used to estimate the time-dependent coefficients of the model. The non-stationary parameters estimated are used as a basis for statistical inference and biophysical interpretation of genomic data. In particular, we rely on the TD-ARMA model of genomic sequences to investigate the statistical properties and differentiate between coding and non-coding regions in the nucleotide chain. Specifically, we define a quantitative measure of randomness to assess how far a process deviates from white noise. Our simulation results on various gene sequences show that both the coding and non-coding regions are non-random. However, coding sequences are "whiter" than non-coding sequences as attested by a higher index of randomness. CONCLUSION: We demonstrate that the proposed TD-ARMA model can be used to provide a stable time series tool for the analysis of non-stationary genomic sequences. The estimated time-varying coefficients are used to define an index of randomness, in order to assess the statistical correlations in coding and non-coding DNA sequences. It turns out that the statistical differences between coding and non-coding sequences are more subtle than previously thought using stationary analysis tools: Both coding and non-coding sequences exhibit statistical correlations, with the coding regions being "whiter" than the non-coding regions. These results corroborate the evolutionary periodogram analysis of genomic sequences and revoke the stationary analysis' conclusion that coding DNA behaves like random sequences.

We develop a general theory of spatially-variant (SV) mathematical morphology for binary images in the Euclidean space. The basic SV morphological operators (i.e., SV erosion, SV dilation, SV opening and SV closing) are defined. We demonstrate the ubiquity of SV morphological operators by providing a SV kernel representation of increasing operators. The latter representation is a generalization of Matheron's representation theorem of increasing and translation-invariant operators. The SV kernel representation is redundant, in the sense that a smaller subset of the SV kernel is sufficient for the representation of increasing operators. We provide sufficient conditions for the existence of the minimal basis representation in terms of upper-semi-continuity in the hit-or-miss topology. The latter minimal basis representation is a generalization of Maragos' minimal basis representation for increasing and translation-invariant operators. Moreover, we investigate the upper-semi-continuity property of the basic SV morphological operators. Several examples are used to demonstrate that the theory of spatially-variant mathematical morphology provides a general framework for the unification of various morphological schemes based on spatiallyvariant geometrical structuring elements (e.g., circular, affine and motion morphology). Simulation results illustrate the theory of the proposed spatially-variant morphological framework and show its potential power in various image processing applications.

In this paper, we develop a spatially-variant (SV) mathematical morphology theory for gray-level signals and images in the Euclidean space. The proposed theory preserves the geometrical concept of the structuring function, which provides the foundation of classical morphology and is essential in signal and image processing applications. We define the basic SV gray-level morphological operators (i.e., SV gray-level erosion, dilation, opening, and closing) and investigate their properties. We demonstrate the ubiquity of SV gray-level morphological systems by deriving a kernel representation for a large class of systems, called V-systems, in terms of the basic SV graylevel morphological operators. A V-system is defined to be a gray-level operator, which is invariant under gray-level (vertical) translations. Particular attention is focused on the class of SV flat gray-level operators. The kernel representation for increasing V-systems is a generalization of Maragos' kernel representation for increasing and translation-invariant function-processing systems. A representation of V-systems in terms of their kernel elements is established for increasing and upper-semi-continuous V-systems. This representation unifies a large class of spatially-variant linear and non-linear systems under the same mathematical framework. Finally, simulation results show the potential power of the general theory of gray-level spatially-variant mathematical morphology in several image analysis and computer vision applications.

I NTEREST from industry and academia has increased dramatically over recent years in the challenging area of event analysis and recognition from various video sources including sports, surveillance, user-generated video, etc. Video event analysis and recognition is a critical task in many applications such as detection of sporting highlights, incident detection in surveillance video, indexing, retrieval and summarization of video databases, and human-computer interaction. This special issue aims to capture the latest advances by the research community working in the area of video event analysis. The call for papers was enthusiastically greeted by the research community and we received over seventy submissions. The special issue presents 16 articles which provide fundamental contributions in a wide range of topics in video event analysis: 1) human action and activity recognition; 2) motion trajectory analysis; 3) video content analysis and pattern mining; 4) audio-visual multi-modal analysis; and 5) video analysis applications. An overview of the organization and a brief summary of the articles selected for publication in the special issue are provided below.

This paper presents a novel particle allocation approach to particle filtering which minimizes the total tracking distortion for a fixed number of particles over a video sequence. We define the tracking distortion as the variance of the error between the true state and estimated state and use rate-distortion theory to determine the optimal particle number and memory size allocation under fixed particle number and memory constraints, respectively. We subsequently provide an algorithm for simultaneous adjustment of the proposal variance and particle number for optimal particle allocation in video tracking systems. Experimental results are used to evaluate the proposed video tracking system and demonstrate its utility for target tracking in numerical examples and video sequences. We demonstrate the superiority of the proposed dynamic proposal variance and optimal particle allocation algorithm in comparison to traditional particle allocation methods, i.e., a fixed number of particles per frame.

In this paper, we propose a maximum-entropy expectation-maximization (MEEM) algorithm. We use the proposed algorithm for density estimation. The maximum-entropy constraint is imposed for smoothness of the estimated density function. The derivation of the MEEM algorithm requires determination of the covariance matrix in the framework of the maximum-entropy likelihood function, which is difficult to solve analytically. We, therefore, derive the MEEM algorithm by optimizing a lower-bound of the maximum-entropy likelihood function. We note that the classical expectation-maximization (EM) algorithm has been employed previously for 2-D density estimation. We propose to extend the use of the classical EM algorithm for image recovery from randomly sampled data and sensor field estimation from randomly scattered sensor networks. We further propose to use our approach in density estimation, image recovery and sensor field estimation. Computer simulation experiments are used to demonstrate the superior performance of the proposed MEEM algorithm in comparison to existing methods.

Many important problems in statistical signal processing can be formulated as function estimation from randomly scattered sensors in a multidimensional space, e.g., image reconstruction from photon-limited images and field estimation from scattered sensors. We present a novel approach to the study of signal reconstruction from random samples in a multidimensional space. In particular, we study a classical iterative reconstruction method and demonstrate that it forms a sequence of unbiased estimates for band-limited signals, which converge to the true function in the mean-square sense. We subsequently rely on the iterative estimation method for multidimensional image reconstruction and field estimation from sensors scattered according to a multidimensional Poisson and uniform distribution. Computer simulation experiments are used to demonstrate the efficiency of the iterative estimation method in image reconstruction and field estimation from randomly scattered sensors.

This correspondence presents a video tracking framework using control-based observer design. It unifies several kernel-based approaches into a consistent theoretical framework by modeling tracking as a recursive inverse problem. The framework relies on observability theory to handle the "singularity" problem and provides explicit criteria for kernel design and dynamics evaluation.

In this paper, we propose a general two-dimensional hidden Markov model (2D-HMM), where dependency of the state transition probability on any state is allowed as long as causality is preserved. The proposed 2D-HMM model can capture, for example, dependency among diagonal states, which can be critical in many image processing applications. A new expectation-maximization (EM) algorithm suitable for estimation of the new model is derived, where a novel general forward-backward (GFB) algorithm is proposed for recursive estimation of the model parameters. A new conditional-independent subset-state sequence structure decomposition is proposed for the 2D Viterbi algorithm. The new model can be applied to many areas such as trajectory classification and image segmentation. Application to aerial image segmentation shows the superiority of our model compared to the existing 2D-HMM model.

In our previous work, we proposed an approach to particle filtering which simultaneously adjusts the proposal variance and number of particles for each frame, in order to minimize the tracking distortion for single object tracking. In this paper, we extend our previous work to multiple object video tracking. Under the framework of distributed multiple object tracking, we propose the tracking distortion and use rate distortion theory to derive the optimal particle allocation among multiple targets as well as multiple frames. We subsequently propose a dynamic proposal variance and optimal particle number allocation algorithm for multi-object tracking. Experimental results show the superior performance of our proposed algorithm to traditional particle allocation methods, i.e., a fixed number of particles for each object in each frame. The proposed algorithm can also be used in decentralized articulated object tracking. To the best of our knowledge, this paper is the first to provide an optimal allocation of a fixed number of particles among multiple objects and frames.

In this paper, we propose a novel technique for estimating focused image sequences captured by an out-of-focus camera. The basic concept used in the proposed algorithm employs multiple images with different blurs from the video sequence to estimate the blur model and construct a more robust method for focused image estimation. This algorithm can be used to correct focusing errors in the video capture process. It could potentially be used to replace the expensive apparatus required for auto-focus adjustments by miniature engines in many camera devices. We demonstrate the successful performance of our approach to focused image estimation through computer simulation experiments using various blur models.

Most existing multiple object tracking techniques suffer from the well-known “multi-object occlusion” problem and/or immense computational cost due to the use of highdimensional joint state representation. In this paper, we present a novel distributed Bayesian framework using multiple collaborative cameras for robust and efficient multiple object tracking with significant and persistent occlusion. When the objects are in close proximity or present multi-object occlusions in a particular camera view, camera collaboration between different views is activated in order to handle the multi-object occlusion problem. Moreover, we propose to model the camera collaboration likelihood density by using epipolar geometry with particle filter implementation. The performance of our approach has been demonstrated on real-world video data.

In this paper, we introduce a method to jointly track the object motion and estimate pose within the framework of particle filtering. We focus on direct estimation of the 3D pose from a 2D image sequence. Scale-Invariant Feature Transform (SIFT) is used to extract feature points in the images. We show that pose estimation from the corresponding feature points can be formed as a solution to Sylvester's equation. We rely on a solution to Sylvester's equation based on the Kronecker product method to solve the equation and determine the pose state. We demonstrate that the classical Singular Value Decomposition (SVD) approach to pose estimation provides a solution to Sylvester's equation in 3D-3D pose estimation. The proposed approach to the solution of Sylvester's equation is therefore equivalent to the classical SVD method for 3D-3D pose estimation, yet it can also be used for pose estimation from 2D image sequences. Finally, we rely on computer simulation experiments to demonstrate the performance of our algorithm on video sequences.

In this paper, we propose a maximum-entropy expectation-maximization algorithm. We use the proposed algorithm for density estimation. The maximum-entropy constraint is imposed in order to ensure smoothness of the estimated density function. The exact derivation of the maximum-entropy expectation-maximization algorithm requires determination of the covariance matrix combined with the maximum entropy likelihood function, which is difficult to solve directly. We therefore introduce a new lower-bound for the EM algorithm derived by using the Cauchy-Schwartz inequality to obtain a suboptimal solution. We use the proposed algorithm for function interpolation and image segmentation. We propose the use of the EM algorithm for image recovery from randomly sampled data and signal reconstruction from randomly scattered sensors. We further propose to use our approach to maximum-entropy expectation-maximization (MEEM) in all of these applications. Computer simulation experiments are used to demonstrate the performance of our algorithm in comparison to existing methods.

In this paper, we propose two novel articulated object tracking approaches. The Decentralized Articulated Object Tracking approach avoids the common practice of using a high-dimensional joint state representation for articulated object tracking. Instead, it introduces a decentralized scheme and models the inter-part interaction within an innovative Bayesian framework. To handle severe self-occlusions, we further extend the first approach by modeling high-level inter-unit interactions and develop the Hierarchical Articulated Object Tracking algorithm within a consistent hierarchical framework. Preliminary experimental results have demonstrated the superior performance of the proposed approaches for real-world videos sequences.

Motion trajectories provide rich spatiotemporal information about an object's activity. This paper presents novel classification algorithms for recognizing object activity using object motion trajectory. In the proposed classification system, trajectories are segmented at points of change in curvature, and the subtrajectories are represented by their principal component analysis (PCA) coefficients. We first present a framework to robustly estimate the multivariate probability density function based on PCA coefficients of the subtrajectories using Gaussian mixture models (GMMs). We show that GMM-based modeling alone cannot capture the temporal relations and ordering between underlying entities. To address this issue, we use hidden Markov models (HMMs) with a data-driven design in terms of number of states and topology (e.g., left-right versus ergodic). Experiments using a database of over 5700 complex trajectories (obtained from UCI-KDD data archives and Columbia University Multimedia Group) subdivided into 85 different classes demonstrate the superiority of our proposed HMM-based scheme using PCA coefficients of subtrajectories in comparison with other techniques in the literature.

In this paper, we present two new articulated motion analysis and object tracking approaches: the decentralized articulated object tracking method and the hierarchical articulated object tracking method. The first approach avoids the common practice of using a high-dimensional joint state representation for articulated object tracking. Instead, we introduce a decentralized scheme and model the interpart interaction within an innovative Bayesian framework. Specifically, we estimate the interaction density by an efficient decomposed interpart interaction model. To handle severe self-occlusions, we further extend the first approach by modeling high-level interunit interaction and develop the second algorithm within a consistent hierarchical framework. Preliminary experimental results have demonstrated the superior performance of the proposed approaches on real-world videos in both robustness and speed compared with other articulated object tracking methods.

This paper presents a novel motion trajectory-based compact indexing and efficient retrieval mechanism for video sequences. Assuming trajectory information is already available, we represent trajectories as temporal ordering of subtrajectories. This approach solves the problem of trajectory representation when only partial trajectory information is available due to occlusion. It is achieved by a hypothesis testing-based method applied to curvature data computed from trajectories. The subtrajectories are then represented by their principal component analysis (PCA) coefficients for optimally compact representation. Different techniques are integrated to index and retrieve subtrajectories, including PCA, spectral clustering, and string matching. We assume a query by example mechanism where an example trajectory is presented to the system and the search system returns a ranked list of most similar items in the dataset. Experiments based on datasets obtained from University of California at Irvine's KDD archives and Columbia University's DVMM group demonstrate the superiority of our proposed PCA-based approaches in terms of indexing and retrieval times and precision recall ratios, when compared to other techniques in the literature

This paper presents a method which avoids the common practice of using a complex joint state-space representation and performing tedious joint data association for multiple object tracking applications. Instead, we propose a distributed Bayesian formulation using multiple interactive trackers that requires much lower complexity for real-time tracking applications. When the objects' observations do not interact with each other, our approach performs as multiple independent trackers. However, when the objects' observations exhibit interaction, defined as close proximity or partial and complete occlusion, we extend the conventional Bayesian tracking framework by modeling such interaction in terms of potential functions. The proposed“magnetic-inertia”model represents the cumulative effect of virtual physical forces that objects undergo while interacting with each other. It implicitly handles the“error merge”and“object labeling”problems and thus solves the difficult object occlusion and data association problems in an innovative way. Our preliminary simulations have demonstrated that the proposed approach is far superior to other methods in both robustness and speed.

This paper presents a novel approach to particle filtering which minimizes the total tracking distortion by considering dynamic variance of proposal density and optimal number of particles for each frame. Traditionally, particle filters use fixed variance of proposal density and fixed number of particles per frame. We propose a tracking distortion measurement and use rate distortion theory to obtain the optimal memory size and particle number allocation equations under different two constraints. We subsequently propose the dynamic proposal variance and optimal particle number allocation algorithm for video tracking systems. Simulation results show the improved performance of our proposed algorithm in comparison to traditional particle filters. To the best of our knowledge, our approach is the first to consider variant numbers of particles for each frame.

Although more efficient in computation compared to other tracking approaches such as particle filtering, the kernel-based tracking suffers from the "singularity" problem which makes the tracking unstable and even completely fail. In this paper, we propose a novel framework to handle this problem by enhancing the tracker's observability. In particular, we formulate object tracking as an inverse problem, thus unifying the existing kernel-based tracking approaches into a consistent theoretical framework. By exploiting the observability theory, we explicitly give the criterion for kernel design and constraint selection. Moreover, we extend the kernel-based approach by including the state dynamics and thus form a state-space model. The use of observability theory is also extended for dynamics estimation and evaluation. We rely on an optimal observer for state estimation as a solution to video tracking. The performance of the proposed approach has been demonstrated on both synthetic and real-world video data and compared to other kernel-based tracking approaches.

Multiple description coding (MDC) combined with multi-path transmission is a viable solution to the growing demand of reliable multimedia video communication over the Internet. Despite the progress in MDC techniques, transport protocols based on TCP cannot efficiently handle multipath transmissions of this kind of traffic. In this paper we present MD-SCTP, a full-multipath partially-reliable SCTP-based protocol, providing different scheduling schemes and featuring a selective retransmission within time-to-delivery constraints of multimedia traffic. Performance analysis proves the validity of our scheduler and of our selective retransmission scheme.

Particle filters have been introduced as a powerful tool to estimate the posterior density of nonlinear systems. These filters are also capable of processing data online as required in many practical applications. In this paper, we propose a novel technique for video stabilization based on the particle filtering framework. Scale-invariant feature points are extracted to form a rough estimate which is used to model the importance density. We use a constant-velocity Kalman filter model to estimate intentional camera movement. We also prove that the particle filtering estimate will lower the error variance. The superior performance and robustness of our algorithm is demonstrated by computer simulations.

This paper presents novel tensor-based object trajectory modelling techniques for simultaneous representation of multiple objects motion trajectories in a content based indexing and retrieval framework. Three different tensor decomposition techniques-PARAFAC, HOSVD and multiple-SVD-are explored to achieve this goal with the aim of using a minimum set of coefficients and data-dependant bases. These tensor decompositions have been applied to represent full as well as segmented trajectories. Our simulation results show that the PARAFAC-based representation provides higher compression ratio, superior precision-recall metrics, and smaller query processing time compared to the other tensor-based approaches

This paper presents a peer-to-peer (P2P) service for the transmission of real-time video content, exploiting the contemporary usage of multiple network paths over the current Internet. Before starting the transmission, the sender probes the available paths for their round trip time and other parameters using simple ICMP packets. Then it chooses the set of paths to be used in order to maximize the expected signal to noise ratio (SNR) while meeting the delay requirements of the streaming content. The video is encoded using a multiple description coding (MDC) technique. Simulations show that the proposed algorithm always chooses one of the group of paths that yields the best SNR

This paper presents a novel approach to particle filtering which minimizes the total tracking distortion by considering dynamic variance of proposal density and adaptive number of particles for each frame. Traditionally, particle filters use fixed variance of proposal density and fixed number of particles per frame. We first propose the tracking distortion measurement and then obtain the optimal particle number and memory size allocation equations under two different constraints. After that, the optimal particle number allocation equation is demonstrated in one-dimensional and two-dimensional object tracking. Experimental results show the improved performance of our power-aware particle filters in comparison to traditional particle filters. At last, we give the complete algorithm for real application and show the better performance. To the best of our knowledge, this paper is the first to consider the variant numbers of particles for each frame

Kernel-based tracking approaches have proven to be more efficient in computation compared to other tracking approaches such as particle filtering. However, existing kernel-based tracking approaches suffer from the well-known "singularity" problem. In this paper, we propose a novel object tracking frame work to handle this problem by using a control-based observer design. Specifically, we formulate object tracking as a recursive inverse problem, thus unifying several approaches to video tracking, including kernel-based tracking, into a consistent theoretical framework. Moreover, we interpret the inverse equation as a measurement process and supplement it by introducing state dynamics as a constraint. The augmented recursive inverse equation forms a state-space model, which is solved by using a control-based optimal observer. By exploiting observability theory from control engineering, we extend the current approach to kernel-based tracking and provide explicit criteria for kernel design and dynamics evaluation. The tracking performance of our approach has been demonstrated on both synthetic and real-world video data

In this paper, we present a novel decentralized Bayesian framework using multiple collaborative cameras for robust and efficient multiple object tracking with significant and persistent occlusion. This approach avoids the common practice of using a complex joint state representation and a centralized processor for multiple camera tracking. When the objects are in close proximity or present multi-object occlusions in a particular camera view, camera collaboration between different views is activated in order to handle the multi-object occlusion problem. Specifically, we propose to model the camera collaboration likelihood density by using epipolar geometry with particle filter implementation. The performance of our approach has been demonstrated on both synthetic and real-world video data

We derive a new method for multidimensional dynamic programming using the inclusion-exclusion principle. We subsequently propose an extension of the Viterbi algorithm to semi-causal, multidimensional functions. This approach is based on extension of the 1-D trellis structure of the Viterbi algorithm to a tree structure in higher dimensions. We apply the dynamic tree programming algorithm to active surface extraction in video sequences. Simulation results show the efficiency and robustness of the proposed approach

Associative processors are single-instruction multiple-data-based architectures that provide huge parallelism which can be very useful for processing video in real time. This paper describes architectural requirements for video and shows that associative processors are ideally suited for applications involving video. Some routines like discrete cosine transforms (DCTs) lack required parallelism and their implementation becomes critical for the overall performance of the processor for video. We present three different approaches for computing two-dimensional DCT on associative processors and discuss their implementation details. Many methods like conditional execution of DCT, have been suggested in literature for power efficiency. In this paper, we discuss integration of these techniques in our implementation. Finally, performance of Associative processors is compared with that of RISC and DSP processor and show that associative processors are better suited for processing video both in terms of speed and power.

The theory of spatially variant (SV) mathematical morphology is used to extend and analyze two important image processing applications: morphological image restoration and skeleton representation of binary images. For morphological image restoration, we propose the SV alternating sequential filters and SV median filters. We establish the relation of SV median filters to the basic SV morphological operators (i.e., SV erosions and SV dilations). For skeleton representation, we present a general framework for the SV morphological skeleton representation of binary images. We study the properties of the SV morphological skeleton representation and derive conditions for its invertibility. We also develop an algorithm for the implementation of the SV morphological skeleton representation of binary images. The latter algorithm is based on the optimal construction of the SV structuring element mapping designed to minimize the cardinality of the SV morphological skeleton representation. Experimental results show the dramatic improvement in the performance of the SV morphological restoration and SV morphological skeleton representation algorithms in comparison to their translation-invariant counterparts.

The particle filtering framework has revolutionized probabilistic tracking of objects in a video sequence. In this framework, the proposal density can be any density as long as its support includes that of the posterior. However, in practice, the number of samples is finite and consequently the choice of the proposal is crucial to the effectiveness of the tracking. The CONDENSATION filter uses the transition prior as the proposal density. We propose in this paper a motion-based proposal. We use adaptive block matching (ABM) as the motion estimation technique. The benefits of this model are two fold. It increases the sampling efficiency and handles abrupt motion changes. Analytically, we derive a Kullback-Leibler (KL)-based performance measure and show that the motion proposal is superior to the proposal of the CONDENSATION filter. Our experiments are applied to head tracking. Finally, we report promising tracking results in complex environments.

We present a novel automatic system integrating head detection with a particle filter for realtime multi-head tracking (MHT) in video. Distinct from the conventional particle filter, which gets particles from the prior density, we propose a novel importance function based on up to date detection and motion observation which makes the particles more effective and helps us to achieve stable tracking by using much fewer particles. We also propose a general likelihood model in the context of MHT. Different information can be fused in a principled manner to make the tracker more stable. The proposed approach can handle not only changes of scale, lighting, zooming, and pose, but also fast motion, appearance, and hard multi-head occlusion.

This paper breaks with the common practice of using a joint state space representation and performing the joint data association in multi-object tracking. Instead, we present an interactively distributed framework with linear complexity for real-time applications. When objects do not interact on each other our approach performs like multiple independent trackers. When, the objects are in close proximity or present occlusions, we propose a magnetic-inertia potential model to handle the "error merge" and "labeling" problems in a particle filtering framework. Specifically we propose to model the interactive likelihood densities by a "gravitation" and "magnetic" repulsion scheme and relax the common first-order Markov chain assumption by using an "inertia" Markov chain. Our model represents the cumulative effect of virtual physical forces that objects undergo while interacting with others. It implicitly handles the "error merge" and "labeling" problems and thus solves the difficult object occlusion and data association problems using an innovative scheme. Our preliminary work has demonstrated that the proposed approach is far superior to existing methods not only in robustness but also in speed.

In this paper, we propose a novel technique for model-based recognition of complex object motion trajectories using hidden Markov models (HMM). We build our models on principal component analysis (PCA)-based representation of trajectories after segmenting them into small units of perceptually similar pieces of motions. These subtrajectories are then grouped using spectral clustering to decide on the number of states for each HMM representing a class of object motion. The hidden states of the HMMs are represented by Gaussian mixtures (GM's). This way the HMM topology as well as the parameters are automatically derived from the training data in a fully unsupervised sense. Experiments are performed on two data sets; the ASL data set (from UCI's KDD archives) consists of 207 trajectories depicting signs for three words, from Australian Sign Language (ASL); the HJSL data set contains 108 trajectories from sports videos. Our experiments yield an accuracy of 90+% performing much better than existing approaches.

In this paper, we propose a novel technique for model-based recognition of complex object motion trajectories using Gaussian mixture models (GMM). We build our models on principal component analysis (PCA)-based representation of trajectories after segmenting them into small units of perceptually similar pieces of motions. These subtrajectories are then fitted with automatically learnt mixture of Gaussians to estimate the underlying class probability distribution. Experiments are performed on two data sets; the ASL data set (from UCI's KDD archives) consists of 207 trajectories depicting signs for three words, from Australian sign language (ASL); the HJSL data set contains 108 trajectories from sports videos. Our experiments yield an accuracy of 85+% performing much better than existing approaches

We present a novel motion trajectory based video retrieval system using LAMSTAR-based adaptive self organizing maps (PASOMs) in this paper. The trajectories are extracted from video by a robust tracker. To reduce the high dimension of motion trajectories, we first decompose each trajectory into sub-trajectories by using a maximum acceleration based approach. Each subtrajectory is then modeled and coded by two different methods, polynomial curving fitting and independent component analysis. To fuse the different features of subtrajectories for more efficient and flexible retrieval, we use PASOMs as the searching tool. Experimental results show the superior performance of the proposed approach for video retrieval comparing with prior approaches.

We develop a new biologically motivated algorithm for representing natural images using successive projections into complementary subspaces. An image is first projected into an edge subspace spanned using an ICA basis adapted to natural images which captures the sharp features of an image like edges and curves. The residual image obtained after extraction of the sharp image features is approximated using a mixture of probabilistic principal component analyzers (MPPCA) model. The model is consistent with cellular, functional, information theoretic, and learning paradigms in visual pathway modeling. We demonstrate the efficiency of our model for representing different attributes of natural images like color and luminance. We compare the performance of our model in terms of quality of representation against commonly used basis, like the discrete cosine transform (DCT), independent component analysis (ICA), and principal components analysis (PCA), based on their entropies. Chrominance and luminance components of images are represented using codes having lower entropy than DCT, ICA, or PCA for similar visual quality. The model attains considerable simplification for learning from images by using a sparse independent code for representing edges and explicitly evaluating probabilities in the residual subspace.

In this paper, the problems of redundancy allocation for providing effective error-resilience and service class distribution for enhanced quality of service (QoS) in real-time MPEG-2 video transport are addressed. A real-time low-complexity content-based adaptive error-resilient approach is proposed for the transport of MPEG-2 video streams, encapsulated using real-time transport protocol (RTP) and delivered over heterogeneous networks. An algorithm is derived using spatial and temporal properties of MPEG-2 video for assigning weights to each packet based on the estimated perceptual error. These weights, which indicate the relative importance of RTP packets, together with the communication channel characteristics are used to determine the allocation of resources for providing improved error-resilience and for assigning data packets to various classes of service in order to enhance the quality of transmission. Parameters extracted from the RTP header are used to determine the weights, so that the proposed algorithm can be implemented in real-time. This algorithm is used for adaptively allocating redundant forward error correction packets as well as for marking and forwarding of RTP packets in differentiated services (DiffServ). Simulation results are presented to show the significant improvement in performance based on our proposed approach to video transport.

We propose a fast object tracking algorithm that predicts the object contour using motion vector information. The segmentation step common in region-based tracking methods is avoided, except for the initialization of the object. Tracking is achieved by predicting the object boundary using block motion vectors followed by updating the contour using occlusions/disocclusion detection. An adaptive block-based approach has been used for estimating motion between frames. An efficient modulation scheme is used to control the gap between frames used for motion estimation. The algorithm for detecting disocclusion proceeds in two steps. First, uncovered regions are estimated from the displaced frame difference. These uncovered regions are classified into actual disocclusions and false alarms by observing the motion characteristics of uncovered regions. Occlusion and disocclusion are considered as dual events and this relationship is explained in detail. The algorithm for detecting occlusion is developed by modifying the disocclusion detection algorithm in accordance with the duality principle. The overall tracking algorithm is computationally superior to existing region-based methods for object tracking. The immediate applications of the proposed tracking algorithm are video compression using MPEG-4 and content retrieval based on standards like H.264. Preliminary simulation results demonstrate the performance of the proposed algorithm.

In this paper, we present a novel principal component analysis (PCA) based approach towards modeling the object trajectory in a video clip. An eigenspace decomposition of high-dimensional trajectory data leads to very compact representation, which is then used as indexing structure. To cutback on PCA computation during indexing, we first segment the trajectories into atomic subtrajectories using a curvature zero-crossing based approach followed by clustering of these subtrajectories. A two-level PCA operation with coarse-to-fine retrieval for query trajectory is then performed to generate retrieval results. Our experimental results show that our global PCA based approach performs better when input query trajectory is of similar length compared to the matching trajectories in the database. However, when partial trajectories are posed as queries our segmented trajectory based approach provides superior results for all precision-recall ratios.

A tracking algorithm that predicts the object contour using motion vector information is proposed in this paper. Tracking is achieved by predicting the object boundary using motion vectors, followed by contour update, using occlusions/disocclusion detection. An adaptive block-based approach has been used for estimating motion between frames. An efficient modulation scheme is used to control the gap between frames used for object tracking. The algorithm for detecting occlusion proceeds in two steps. First, covered regions are estimated from the displaced frame difference. These covered regions are classified into actual occlusions and false alarms using motion characteristics. Disocclusion detection is also performed in a similar manner. The immediate applications of the proposed tracking algorithm are video compression using MPEG-4 and content retrieval based on standards like H.26L. Preliminary simulation results demonstrate the performance of the proposed algorithm.

We present an algorithm for real-time adaptive forward error correction (FEC) of MPEG-2 video stream, encapsulated using real-time transport protocol (RTP) and delivered over best-effort networks. Our algorithm provides an efficient method to determine the allocation of redundancy to the MPEG-2 video stream. The redundancy is allocated such that the resulting estimated degradation density function for video (DDF) is uniformly distributed. A weight, which indicates the relative importance of RTP packets, together with the communication channel characteristics and FEC scheme are used to model the density function of the video stream and allow us to determine the allocation of FEC packets. The weight is based on the content of RTP packets in the video stream. Parameters extracted from the RTP header are used to determine the weights, so that the proposed algorithm can be implemented in real-time. In our simulations, we have relied on motion compensation and group of picture (GOP) data to determine the relative weights. Simulation results provided establish the significant improvement in performance based on our proposed approach to adaptive FEC.

We propose an object-tracking algorithm that predicts the object contour using motion vector information. Tracking is achieved by predicting the object boundary using motion vectors, followed by contour update, using occlusions/disocclusion detection. An adaptive block-based approach has been used for estimating motion between frames. An efficient modulation scheme is used to control the gap between frames used for object tracking. The algorithm for detecting occlusions proceeds in two steps. First, covered regions are estimated from the displaced frame difference. These covered regions are classified into actual occlusions and false alarms using motion characteristics. Disocclusion detection is also performed in a similar manner. The immediate applications of the proposed tracking algorithm are video compression using MPEG-4 and content retrieval based on standards like H.26L. Preliminary simulation results demonstrate the performance of the proposed algorithm.

The increased availability and usage of multimedia information have created a critical need for efficient multimedia processing algorithms. These algorithms must offer capabilities related to browsing, indexing, and retrieval of relevant data. A crucial step in multimedia processing is that of reliable video segmentation into visually coherent video shots through scene change detection. Video segmentation enables subsequent processing operations on video shots, such as video indexing, semantic representation, or tracking of selected video information. Since video sequences generally contain both abrupt and gradual scene changes, video segmentation algorithms must be able to detect a large variety of changes. While existing algorithms perform relatively well for detecting abrupt transitions (video cuts), reliable detection of gradual changes is much more difficult. A novel one-pass, real-time approach to video scene change detection based on statistical sequential analysis and operating on a compressed multimedia bitstream is proposed. Our approach models video sequences as stochastic processes, with scene changes being reflected by changes in the characteristics (parameters) of the process. Statistical sequential analysis is used to provide an unified framework for the detection of both abrupt and gradual scene changes.

A new streaming protocol is proposed for multimedia applications. The proposed protocol, referred to as Lightweight Streaming Protocol (LSP), is an application layer protocol that sits atop UDP. The protocol is intended to improve the quality and reliability of media stream by borrowing features from reliable protocols such as retransmission and flow control while not sacrificing performance. The protocol offers semi-reliable transport. Instead of trying to guarantee 100% data delivery the protocol simply recovers as many packets as possible within a specified deadline. In addition the protocol incorporates features such as probabilistic redundant NAK transmission and flow control through selective frame dropping. Preliminary simulations show that LSP performs extremely well in channels with random packet loss such as congested networks. The protocol also performs reasonably well in channels that have short bursts of lost packets such as wireless networks.

In this paper, we investigate methods for optimal morphological pattern recognition. The task of optimal pattern recognition is posed as a solution to a hypothesis testing problem. A minimum probability of error decision rule-maximum a posteriori filter-is sought. The classical solution to the minimum probability of error hypothesis testing problem, in the presence of independent and identically distributed noise degradation, is provided by template matching (TM). A modification of this task, seeking a solution to the minimum probability of error hypothesis testing problem, in the presence of composite (mixed) independent and identically distributed noise degradation, is demonstrated to be given by weighted composite template matching (WCTM). As a consequence of our investigation, the relationship of the order-statistics filter (OSF) and TM-in both the standard as well as the weighted and composite implementations-is established. This relationship is based on the thresholded cross-correlation representation of the OSF. The optimal order and weights of the OSF for pattern recognition are subsequently derived. An additional outcome of this representation is a fast method for the implementation of the OSF.

In this paper, we investigate the relation between the image restoration and the template matching problems. We present a new formulation for template matching that establishes the duality between these two problems. According to this formulation the restored template is used to ascertain its location. An alternative formulation is also suggested which is based on the observation that template matching can be viewed as an image restoration problem where the degraded image is blurred by a linear point spread function-the template. It is shown that, unlike image restoration, all second order statistics required for the implementation of linear minimum mean-square error (LMMSE) template matching can be accurately estimated. LMMSE template matching is subsequently applied to the block-matching based motion estimation from degraded image sequences. Our experimental results indicate that LMMSE template matching is superior to traditional block-matching.

The authors comments that Charif-Chefchaouni and Schonfeld (see ibid., vol.3, no.6, p.847, 1994) investigated the invertibility of a morphological representation of binary images and determined the necessary and sufficient conditions for its inverse. The authors show that one of the derived necessary conditions is not valid. A counterexample is given to illustrate our observations. Charif-Chefchaouni and Schonfeld reply that the new sufficient condition is proposed for the invertibility of the morphological image representation. A modification of its inverse is subsequently used to derive a new necessary condition for the invertibility of the morphological image representation. A composition of these conditions is finally used to provide a new necessary and sufficient condition under some restrictions for the invertibility of the morphological image representation. These necessary and sufficient conditions form a revision of one of the necessary conditions for the invertibility of the morphological image representation stated in the original paper.

The method of projections onto convex sets has been used effectively in the solution many important signal and image processing applications. This method however fails in circumstances when the desired constraints form non-convex sets. A new approach to image recovery based on a stochastic projections method is presented. At its core the method relies on the iteration of projections onto closed sets (not necessarily convex). A stochastic parameter is used to update the projections among the images in their neighborhood subsequent to each iteration. Computer simulation experiments are used for the comparison of the stochastic projections with several existing methods to signal recovery for phase-from-magnitude image restoration.

We propose a comprehensive theory for the morphological bounds on order-statistics filters (and their repeated iterations). Conditions are derived for morphological openings and closings to serve as bounds (lower and upper, respectively) on order-statistics filters (and their repeated iterations). Under various assumptions, morphological open-closings and close-openings are also shown to serve as (tighter) bounds (lower and upper, respectively) on iterations of order-statistics filters. Simulations of the application of the results presented to image restoration are finally provided.

In this paper, we present a comprehensive theory of spatially-variant (SV) mathematical morphology. A kernel representation of increasing operators in terms of the union (resp., intersection) of SV erosions (resp., SV dilations) is provided. A representation of algebraic openings (resp., algebraic closings) in terms of the union (resp., intersection) of SV openings (resp., SV closings) is also provided.>

In this paper, we derive the optimal structuring elements of morphological filters in image restoration. The expected pattern transformation of random sets is presented. An estimation theory framework for random sets is subsequently proposed. This framework is based on the least mean difference (LMD) estimator. The LMD estimator is defined to minimize the cardinality of the expected pattern transformation of the set-difference of the parameter and the estimate. Several important results for the determination of the LMD estimator are derived. The LMD structuring elements of morphological filters in image restoration are finally derived.>

We investigate the invertibility of the morphological representation of binary images. A criteria for the invertibility of the morphological representation of binary images is proposed. Necessary and sufficient conditions for the exact reconstruction of a binary image from its morphological representation are provided.

In this correspondence, we present a fast thresholded linear convolution representation of morphological operations. The thresholded linear convolution representation of dilation and erosion is first proposed. A comparison of the efficiency of the direct implementation of morphological operations and the thresholded linear convolution representation of morphological operations is subsequently conducted. Mathematical morphology has emerged as a powerful new tool for image processing.

A mathematical framework for the solution of statistical inference problems on a class of random sets is proposed. It is based on a new definition of expected pattern. The least-mean-difference estimator (restoration filter) is proved, under certain conditions, to be equivalent to the minimization of the measure of size (area) of the set-difference between the original pattern and the expected pattern of the estimated (restored) pattern. Consequently, it is proved that, under certain conditions, if the estimator (restoration filter) is unbiased, then it is the least mean difference estimator (restoration filter).>

The discrete representation of continuous linear time-invariant signal processing systems is discussed. The orthogonal Huggins representation is introduced as an important special case of the discrete orthogonal representation. This representation uses an orthogonalized collection of complex exponentials as the orthogonal basis. Additionally, the Huggins representation of continuous linear time-invariant systems is proposed. The Huggins representation is formed as the discrete eigenfunction representation of continuous linear time-invariant systems in terms of a collection of complex exponentials. The orthogonal Huggins representation results in a quadratic computational complexity, whereas the Huggins representation yields a linear computational complexity.>

A theoretical analysis of morphological filters for the optimal restoration of noisy binary images is presented. The problem is formulated in a general form, and an optimal solution is obtained by using fundamental tools from mathematical morphology and decision theory. Consideration is given to the set-difference distance function as a measure of comparison between images. This function is used to introduce the mean-difference function as a quantitative measure of the degree of geometrical and topological distortion introduced by morphological filtering. It is proved that the class of alternating sequential filters is a set of parametric, smoothing morphological filters that best preserve the crucial structure of input images in the least-mean-difference sense.>

A general theory for the morphological representation of discrete and binary images is presented. Particular cases of the general scheme are shown to yield a number of useful image representations. The effect of noise degradation is studied. It is proven that, under certain assumptions, the general reduced morphological skeleton is the best morphological representation among a collection of invertible morphological image representations. This representation results in a minimal upper-bound on the average probability of error of reconstructing a binary image from its noisy representation.>

A general theory for the morphological representation of discrete and binary images is presented. The theory relies on the generation of a set of nonoverlapping segments of an image by repeated erosions and set transformations, which in turn produces a decomposition that guarantees exact reconstruction. The morphological image-representation transform and its properties are examined, with focus on the relationship between the transform and some existing shape-analysis tools, thus introducing the transform as the basis of a unified geometrical image analysis. Particular cases of the general representation scheme are shown to yield a number of useful image representations which are directly related to various forms of digital morphological skeletons. Also considered is the relationship between the transform and the various forms of morphological skeletons.>