We propose a simple but effective tracking-by-segmentation algorithm using Absorbing Markov Chain (AMC) on superpixel segmentation, where target state is estimated by a combination of bottom-up and top-down approaches, and target segmentation is propagated to subsequent frames in a recursive manner. Our algorithm constructs a graph for AMC using the superpixels identified in two consecutive frames, where background superpixels in the previous frame correspond to absorbing vertices while all other superpixels create transient ones. The weight of each edge depends on the similarity of scores in the end superpixels, which are learned by support vector regression. Once graph construction is completed, target segmentation is estimated using the absorption time of each superpixel. The proposed tracking algorithm achieves substantially improved performance compared to the state-of-the-art segmentation-based tracking techniques in multiple challenging datasets.

We propose a simple but effective unsupervised learning algorithm to detect a common activity (co-activity) from a set of videos, which is formulated using absorbing Markov chain in a principled way. In our algorithm, a complete multipartite graph is first constructed, where vertices correspond to subsequences extracted from videos using a temporal sliding window and edges connect between the vertices originated from different videos; the weight of an edge is proportional to the similarity between the features of two end vertices. Then, we extend the graph structure by adding edges between temporally overlapped subsequences in a video to handle variable-length co-activities using temporal locality, and create an absorbing vertex connected from all other nodes. The proposed algorithm identifies a subset of subsequences as co-activity by estimating absorption time in the constructed graph efficiently. The great advantage of our algorithm lies in the properties that it can handle more than two videos naturally and identify multiple instances of a co-activity with variable lengths in a video. Our algorithm is evaluated intensively in a challenging dataset and illustrates outstanding performance quantitatively and qualitatively.

We present a novel approach in describing and detecting the composite video events based on scenarios, which constrain the configurations of target events by temporal-logical structures of primitive events. We propose a new scenario description method to represent composite events more fluently and efficiently, and discuss an on-line event detection algorithm based on a combinatorial optimization. For this purpose, constraint flow-a dynamic configuration of scenario constraints-is first generated automatically by our scenario parsing algorithm. Then, composite event detection is formulated by a constrained discrete optimization problem, whose objective is to find the best video interpretation with respect to the constraint flow. Although the search space for the optimization problem is prohibitively large, our on-line event detection algorithm based on constraint flow using dynamic programming reduces the search space dramatically, handles preprocessing errors effectively, and guarantees a globally optimal solution. Experimental results on natural videos demonstrate the effectiveness of our algorithm.

We present a joint estimation technique of event localization and role assignment when the target video event is described by a scenario. Specifically, to detect multi-agent events from video, our algorithm identifies agents involved in an event and assigns roles to the participating agents. Instead of iterating through all possible agent-role combinations, we formulate the joint optimization problem as two efficient sub problems-quadratic programming for role assignment followed by linear programming for event localization. Additionally, we reduce the computational complexity significantly by applying role-specific event detectors to each agent independently. We test the performance of our algorithm in natural videos, which contain multiple target events and nonparticipating agents.

We propose an on-line algorithm to extract a human by foreground/background segmentation and estimate pose of the human from the videos captured by moving cameras. We claim that a virtuous cycle can be created by appropriate interactions between the two modules to solve individual problems. This joint estimation problem is divided into two sub problems, foreground/background segmentation and pose tracking, which alternate iteratively for optimization, segmentation step generates foreground mask for human pose tracking, and human pose tracking step provides fore-ground response map for segmentation. The final solution is obtained when the iterative procedure converges. We evaluate our algorithm quantitatively and qualitatively in real videos involving various challenges, and present its outstanding performance compared to the state-of-the-art techniques for segmentation and pose estimation.

We present a novel approach to representing and recognizing composite video events. A composite event is specified by a scenario, which is based on primitive events and their temporal-logical relations, to constrain the arrangements of the primitive events in the composite event. We propose a new scenario description method to represent composite events fluently and efficiently. A composite event is recognized by a constrained optimization algorithm whose constraints are defined by the scenario. The dynamic configuration of the scenario constraints is represented with constraint flow, which is generated from scenario automatically by our scenario parsing algorithm. The constraint flow reduces the search space dramatically, alleviates the effect of preprocessing errors, and guarantees the globally optimal solution for recognition. We validate our method to describe scenario and construct constraint flow for real videos and illustrate the effectiveness of our composite event recognition algorithm for natural video events.

We propose a novel background subtraction algorithm for the videos captured by a moving camera. In our technique, foreground and background appearance models in each frame are constructed and propagated sequentially by Bayesian filtering. We estimate the posterior of appearance, which is computed by the product of the image likelihood in the current frame and the prior appearance propagated from the previous frame. The motion, which transfers the previous appearance models to the current frame, is estimated by nonparametric belief propagation; the initial motion field is obtained by optical flow and noisy and incomplete motions are corrected effectively through the inference procedure. Our framework is represented by a graphical model, where the sequential inference of motion and appearance is performed by the combination of belief propagation and Bayesian filtering. We compare our algorithm with the existing state-of-the-art technique and evaluate its performance quantitatively and qualitatively in several challenging videos.

We propose a novel algorithm to detect occlusion for visual tracking through learning with observation likelihoods. In our technique, target is divided into regular grid cells and the state of occlusion is determined for each cell using a classifier. Each cell in the target is associated with many small patches, and the patch likelihoods observed during tracking construct a feature vector, which is used for classification. Since the occlusion is learned with patch likelihoods instead of patches themselves, the classifier is universally applicable to any videos or objects for occlusion reasoning. Our occlusion detection algorithm has decent performance in accuracy, which is sufficient to improve tracking performance significantly. The proposed algorithm can be combined with many generic tracking methods, and we adopt L1 minimization tracker to test the performance of our framework. The advantage of our algorithm is supported by quantitative and qualitative evaluation, and successful tracking and occlusion reasoning results are illustrated in many challenging video sequences.

We present a new method for segmenting the foreground region of the image of multiple pedestrians from monocular surveillance videos. This method requires neither camera calibration nor planar ground assumption. The size and orientation of a pedestrian projection are estimated at each image point and registered in a pedestrian projection map. Individual pedestrians are segmented from the foreground region of input images using an expectation maximization (EM) algorithm and the constructed map.

Recognition of events in video is an important subject in intelligent video surveillance. In this paper, we propose a new paradigm of event recognition scheme from video. In this structure, most video events are represented by a hierarchical structure, efficient events representation and analysis of events are possible by using this property. We introduce a scalable and hierarchical event recognition method. First, events are classified into four hierarchical categories. Higher level events are organized by lower level events and relationships among them. We represent those relationships using temporal-logical constraints, that is, the event grammar, and a dynamic Bayesian network (DBN) combines the given event grammar with the probabilistic inference procedure to recognize an event. For scalability of the recognition system, all events in the hierarchy use the same framework of DBN. To recognize events efficiently in such a condition, we define the activation rate which is calculated by each event and propagated in bottom-up direction at each time step. We apply the proposed method to the experiments with a video segment simulating ticket office transactions.

Matching features computed in images is an important process in multiview image analysis. When the motion between two images is large, the matching problem becomes very difficult. In this paper, we propose a contour matching algorithm based on geometric constraints. With the assumption that the contours are obtained from images taken from a moving camera with static scenes, we apply the epipolar constraint between two sets of contours and compute the corresponding points on the contours. From the initial epipolar constraints obtained from corner point matching, candidate contours are selected according to the epipolar geometry, contour end point constraints, and contour distance measures. In order to reduce the possibility of false matches, the number of match points on a contour is also used as a selection measure. The initial epipolar constraint is refined from the matched sets of contours. The algorithm can be applied to a pair or two pairs of images. All of the processes are fully automatic and successfully implemented and tested with various real images.

We present a novel method of velocity field estimation for points on moving contours in an image sequence. The method determines the corresponding point in the next image frame by considering curvature changes at each point on a contour. In previous methods, there are errors in estimation for the points which have low curvature variations since those methods compute the solutions by approximating the normal component of optical flow. The proposed method computes optical flow vectors of contour points by minimizing the curvature changes. As a first step, snakes are used to locate smooth curves in 2D imagery. Then, the extracted curves are tracked continuously. We excluded the rearranging process in snakes and allowed the snaxel distance to vary. Each point on a contour has a unique corresponding point in the nest frame. Experimental results showed that the proposed method computes accurate optical flow vectors for various moving contours.

The authors present a method, called distance accumulation, of computing features of closed planar boundaries of 2-D digital images, or closed curves. Distance accumulation is computed by accumulating the distance from a point in the boundary to a chord specified by moving end points. Experimental results with simulated and real images showed its robustness. The analysis of its relation to planar curvature matches experimental results well. >