Fault diagnosis in industrial facilities has traditionally been done using rule-based approaches, heuristics or expert-knowledge. Bayesian network provides a flexible and data-driven alternative that can reason under uncertainty. Most of the data being generated by sensors in industrial setups are continuous and the underlying data-generating models are essentially non-linear. This paper employs Bayesian network and proposes a framework that learns parameters of probability density functions of a continuous Bayesian network using neural network/s without requiring assumption of linear Gaussian model or discretization of continuous data. Moreover, an expression of probability query using learned parametric density functions and causal-inference based mathematical formulation of two tasks related to fault diagnosis –in the context of industrial plants– namely root-cause-analysis and identification of most-influential-path in Bayesian network have been provided.

In this paper a notion of flow complexity that measures the amount of interaction among objects is introduced and an approach to compute it directly from a video sequence is proposed. The approach employs particle trajectories as the input representation of motion and maps it into a `braid' based representation. The mapping is based on the observation that 2D trajectories of particles take the form of a braid in space-time due to the intermingling among particles over time. As a result of this mapping, the problem of estimating the flow complexity from particle trajectories becomes the problem of estimating braid complexity, which in turn can be computed by measuring the topological entropy of a braid. For this purpose recently developed mathematical tools from braid theory are employed which allow rapid computation of topological entropy of braids. The approach is evaluated on a dataset consisting of open source videos depicting variations in terms of types of moving objects, scene layout, camera view angle, motion patterns, and object densities. The results show that the proposed approach is able to quantify the complexity of the flow, and at the same time provides useful insights about the sources of the complexity.

We propose to use action, scene and object concepts as semantic attributes for classification of video events in InTheWild content, such as YouTube videos. We model events using a variety of complementary semantic attribute features developed in a semantic concept space. Our contribution is to systematically demonstrate the advantages of this concept-based event representation (CBER) in applications of video event classification and understanding. Specifically, CBER has better generalization capability, which enables to recognize events with a few training examples. In addition, CBER makes it possible to recognize a novel event without training examples (i.e., zero-shot learning). We further show our proposed enhanced event model can further improve the zero-shot learning. Furthermore, CBER provides a straightforward way for event recounting/understanding. We use the TRECVID Multimedia Event Detection (MED11) open source event definitions and datasets as our test bed and show results on over 1400 hours of videos.

Low-level appearance as well as spatio-temporal features, appropriately quantized and aggregated into Bag-of-Words (BoW) descriptors, have been shown to be effective in many detection and recognition tasks. However, their effcacy for complex event recognition in unconstrained videos have not been systematically evaluated. In this paper, we use the NIST TRECVID Multimedia Event Detection (MED11 [1]) open source dataset, containing annotated data for 15 high-level events, as the standardized test bed for evaluating the low-level features. This dataset contains a large number of user-generated video clips. We consider 7 different low-level features, both static and dynamic, using BoW descriptors within an SVM approach for event detection. We present performance results on the 15 MED11 events for each of the features as well as their combinations using a number of early and late fusion strategies and discuss their strengths and limitations.

Visual landmark matching with a pre-built landmark database is a popular technique for localization. Traditionally, landmark database was built with visual odometry system, and the 3D information of each visual landmark is reconstructed from video. Due to the drift of the visual odometry system, a global consistent landmark database is difficult to build, and the inaccuracy of each 3D landmark limits the performance of landmark matching. In this paper, we demonstrated that with the use of precise 3D Li-dar range data, we are able to build a global consistent database of high precision 3D visual landmarks, which improves the landmark matching accuracy dramatically. In order to further improve the accuracy and robustness, landmark matching is fused with a multi-stereo based visual odometry system to estimate the camera pose in two aspects. First, a local visual odometry trajectory based consistency check is performed to reject some bad landmark matchings or those with large errors, and then a kalman filtering is used to further smooth out some landmark matching errors. Finally, a disk-cache-mechanism is proposed to obtain the real-time performance when the size of the landmark grows for a large-scale area. A week-long real time live marine training experiments have demonstrated the high-precision and robustness of our proposed system.

It has been argued in recent work that correlation filters are attractive for human action recognition from videos. Motivation for their employment in this classification task lies in their ability to: (i) specify where the filter should peak in contrast to all other shifts in space and time, (ii) have some degree of tolerance to noise and intra-class variation (allowing learning from multiple examples), and (iii) can be computed deterministically with low computational overhead. Specifically, Maximum Average Correlation Height (MACH) filters have exhibited encouraging results~\cite{Mikel} on a variety of human action datasets. Here, we challenge the utility of correlation filters, like the MACH filter, in these circumstances. First, we demonstrate empirically that identical performance can be attained to the MACH filter by simply taking the~\emph{average} of the same action specific training examples. Second, we characterize theoretically and empirically under what circumstances a MACH filter would become equivalent to the average of the action specific training examples. Based on this characterization, we offer an alternative type of filter, based on a discriminative paradigm, that circumvent the inherent limitations of correlation filters for action recognition and demonstrate improved action recognition performance.

We propose a framework to learn scene semantics from surveillance videos. Using the learnt scene semantics, a video analyst can efficiently and effectively retrieve the hidden semantic relationship between homogeneous and heterogeneous entities existing in the surveillance system. For learning scene semantics, the algorithm treats different entities as nodes in a graph, where weighted edges between the nodes represent the "initial" strength of the relationship between entities. The graph is then embedded into a k-dimensional space by Fiedler Embedding.

We address the problem of interactive search for a target of interest in surveillance imagery. Our solution consists of iteratively learning a distance metric for retrieval, based on user feedback. The approach employs (retrieval) rank based constraints and convex optimization to efficiently learn the distance metric. The algorithm uses both user labeled and unlabeled examples in the learning process. The method is fast enough for a new metric to be learned interactively for each target query. In order to reduce the burden on the user, a model-independent active learning method is used to select key examples, for response solicitation. This leads to a significant reduction in the number of user-interactions required for retrieving the target of interest. The proposed method is evaluated on challenging pedestrian and vehicle data sets, and compares favorably to the state of the art in target re-acquisition algorithms.

We propose a set of kinematic features that are derived from the optical flow for human action recognition in videos. The set of kinematic features includes divergence, vorticity, symmetric and antisymmetric flow fields, second and third principal invariants of flow gradient and rate of strain tensor, and third principal invariant of rate of rotation tensor. Each kinematic feature, when computed from the optical flow of a sequence of images, gives rise to a spatiotemporal pattern. It is then assumed that the representative dynamics of the optical flow are captured by these spatiotemporal patterns in the form of dominant kinematic trends or kinematic modes. These kinematic modes are computed by performing Principal Component Analysis (PCA) on the spatiotemporal volumes of the kinematic features. For classification, we propose the use of multiple instance learning (MIL) in which each action video is represented by a bag of kinematic modes. Each video is then embedded into a kinematic-mode-based feature space and the coordinates of the video in that space are used for classification using the nearest neighbor algorithm. The qualitative and quantitative results are reported on the benchmark data sets.

This paper presents a target tracking framework for unstructured crowded scenes. Unstructured crowded scenes are defined as those scenes where the motion of a crowd appears to be random with different participants moving in different directions over time. This means each spatial location in such scenes supports more than one, or multi-modal, crowd behavior. The case of tracking in structured crowded scenes, where the crowd moves coherently in a common direction, and the direction of motion does not vary over time, was previously handled in. In this work, we propose to model various crowd behavior (or motion) modalities at different locations of the scene by employing Correlated Topic Model (CTM) of. In our construction, words correspond to low level quantized motion features and topics correspond to crowd behaviors. It is then assumed that motion at each location in an unstructured crowd scene is generated by a set of behavior proportions, where behaviors represent distributions over low-level motion features. This way any one location in the scene may support multiple crowd behavior modalities and can be used as prior information for tracking. Our approach enables us to model a diverse set of unstructured crowd domains, which range from cluttered time-lapse microscopy videos of cell populations in vitro, to footage of crowded sporting events.

In this paper, we propose a framework that fuses multiple features for improved action recognition in videos. The fusion of multiple features is important for recognizing actions as often a single feature based representation is not enough to capture the imaging variations (view-point, illumination etc.) and attributes of individuals (size, age, gender etc.). Hence, we use two types of features: i) a quantized vocabulary of local spatio-temporal (ST) volumes (or cuboids), and ii) a quantized vocabulary of spin-images, which aims to capture the shape deformation of the actor by considering actions as 3D objects (x, y, t). To optimally combine these features, we treat different features as nodes in a graph, where weighted edges between the nodes represent the strength of the relationship between entities. The graph is then embedded into a k-dimensional space subject to the criteria that similar nodes have Euclidian coordinates which are closer to each other. This is achieved by converting this constraint into a minimization problem whose solution is the eigenvectors of the graph Laplacian matrix. This procedure is known as Fiedler Embedding. The performance of the proposed framework is tested on publicly available data sets. The results demonstrate that fusion of multiple features helps in achieving improved performance, and allows retrieval of meaningful features and videos from the embedding space.

Learning dominant motion patterns or activities from a video is an important surveillance problem, especially in crowded environments like markets, subways etc., where tracking of individual objects is hard if not impossible. In this paper, we propose an algorithm that uses instantaneous motion field of the video instead of long-term motion tracks for learning the motion patterns. The motion field is a collection of independent flow vectors detected in each frame of the video where each flow is vector is associated with a spatial location. A motion pattern is then defined as a group of flow vectors that are part of the same physical process or motion pattern. Algorithmically, this is accomplished by first detecting the representative modes (sinks) of the motion patterns, followed by construction of super tracks, which are the collective representation of the discovered motion patterns. We also use the super tracks for event-based video matching. The efficacy of the approach is demonstrated on challenging real-world sequences.

Learning typical motion patterns or activities from videos of crowded scenes is an important visual surveillance problem. To detect typical motion patterns in crowded scenarios, we propose a new method which utilizes the instantaneous motions of a video, i.e, the motion flow field, instead of long-term motion tracks. The motion flow field is a union of independent flow vectors computed in different frames. Detecting motion patterns in this flow field can therefore be formulated as a clustering problem of the motion flow fields, where each motion pattern consists of a group of flow vectors participating in the same process or motion. We first construct a directed neighborhood graph to measure the closeness of flow vectors. A hierarchical agglomerative clustering algorithm is applied to group flow vectors into desired motion patterns.

In this paper, we use motion and appearance contexts for persistent tracking of objects in aerial imagery. The motion context in a given environment is a collection of trajectories of objects which are representative of the motion of the occluded or unobserved object. It is learned using a clustering scheme based on the Lyapunov characteristic exponent (LCE) which measures the mean exponential rate of divergence of the nearby trajectories. The learned motion context is then used in a regression framework to predict the location of the unobserved object. The appearance context of an occluded (target) object consists of appearance information of objects which are currently occluded or unobserved. It is incorporated by learning a distribution of interclass variation for each target-unobservable object pair. In addition, intra-class variation distribution is constructed for each occluded object using all of its previous observations. Qualitative and quantitative results are reported on challenging aerial sequences.

This paper proposes a framework in which Lagrangian particle dynamics is used for the segmentation of high density crowd flows and detection of flow instabilities. For this purpose, a flow field generated by a moving crowd is treated as an aperiodic dynamical system. A grid of particles is overlaid on the flow field, and is advected using a numerical integration scheme. The evolution of particles through the flow is tracked using a flow map, whose spatial gradients are subsequently used to setup a Cauchy Green deformation tensor for quantifying the amount by which the neighboring particles have diverged over the length of the integration. The maximum eigenvalue of the tensor is used to construct a finite time Lyapunov exponent (FTLE) field, which reveals the Lagrangian coherent structures (LCS) present in the underlying flow. The LCS divide flow into regions of qualitatively different dynamics and are used to locate boundaries of the flow segments in a normalized cuts framework. Any change in the number of flow segments over time is regarded as an instability, which is detected by establishing correspondences between flow segments over time. The experiments are conducted on a challenging set of videos taken from Google Video and a National Geographic documentary.

The paper introduces an action recognition framework that uses concepts from the theory of chaotic systems to model and analyze nonlinear dynamics of human actions. Trajectories of reference joints are used as the representation of the non-linear dynamical system that is generating the action. Each trajectory is then used to reconstruct a phase space of appropriate dimension by employing a delay-embedding scheme. The properties of the reconstructed phase space are captured in terms of dynamical and metric invariants that include Lyapunov exponent, correlation integral and correlation dimension. Finally, the action is represented by a feature vector which is a combination of these invariants over all the reference trajectories. Our contributions in this paper include :1) investigation of the appropriateness of theory of chaotic systems for human action modelling and recognition, 2) a new set of features to characterize nonlinear dynamics of human actions, 3) experimental validation of the feasibility and potential merits of carrying out action recognition using methods from theory of chaotic systems.

In this paper we introduce a 3-dimensional (3D) SIFT descriptor for video or 3D imagery such as MRI data. We also show how this new descriptor is able to better represent the 3D nature of video data in the application of action recognition. This paper will show how 3D SIFT is able to outperform previously used description methods in an elegant and efficient manner. We use a bag of words approach to represent videos, and present a method to discover relationships between spatio-temporal words in order to better describe the video data.

Boosting based detection methods have successfully been used for robust detection of faces and pedestrians. However, a very large amount of labeled examples are required for training such a classifier. Moreover, once trained, the boosted classifier cannot adjust to the particular scenario in which it is employed. In this paper, we propose a co-training based approach to continuously label incoming data and use it for online update of the boosted classifier that was initially trained from a small labeled example set. The main contribution of our approach is that it is an online procedure in which separate views (features) of the data are used for co-training, while the combined view (all features) is used to make classification decisions in a single boosted framework. The features used for classification are derived from principal component analysis of the appearance templates of the training examples. In order to speed up the classification, background modeling is used to prune away stationary regions in an image. Our experiments indicate that starting from a classifier trained on a small training set, significant performance gains can be made through online updation from the unlabeled data.

In recent years kernel principal component analysis (kernel PCA) has gained much attention because of its ability to capture nonlinear image features, which are particularly important for encoding image structure. Boosting has been established as a powerful learning algorithm that can be used for feature selection. In this paper we present a novel framework for object class detection that combines the feature reduction and feature selection abilities of kernel PCA and AdaBoost respectively. The classifier obtained in this way is able to handle change in object appearance, illumination conditions, and surrounding clutter. A nonlinear subspace is learned for positive and negative object classes using Kernel PCA. Features are derived by projecting example images onto the learned subspaces. Base learners are modeled using Bayes classifier. AdaBoost is then employed to discover the features that are most relevant for the object detection task at hand. The proposed method has been successfully tested on wide range of object classes (cars, air-planes, pedestrians, motorcycles, etc) using standard data sets and has shown remarkable performance. Using a small training set, a classifier learned in this way was able to generalize the intra-class variation while still maintaining high detection rate. In most object categories we achieved detection rates of above 95% with minimal false alarm rates. We demonstrate the effectiveness of our approach in terms of absolute performance parameters and comparative performance against current state of the art approaches.

In this paper, we present a novel framework for generic object class detection by integrating Kernel PCA with AdaBoost. The classifier obtained in this way is invariant to changes in appearance, illumination conditions and surrounding clutter. A nonlinear shape subspace is learned for positive and negative object classes using kernel PCA. Features are derived by projecting example images onto the learned sub-spaces. Base learners are modeled using Bayes classifier. AdaBoost is then employed to discover the features that are most relevant for the object detection task at hand. Proposed method has been successfully tested on wide range of object classes (cars, airplanes, pedestrians, motorcycles etc) using standard data sets and has shown good performance. Using a small training set, the classifier learned in this way was able to generalize the intra-class variation while still maintaining high detection rate. In most object categories, we achieved detection rates of above 95% with minimal false alarm rates. We demonstrate the comparative performance of our method against current state of the art approaches.