We propose a deep learning framework to analyse the uncertainty associated with dyadic human activities at a small temporal granularity. Such time-slice analysis is able to infer human behaviours from short-term observations. Instead of classifying time-slices into k classes of activities, we report to what degree of certainty each activity is occurring from definitely not occurring to definitely occurring. To this end, we extract CNN-based unary probabilities and pairwise relations between body joints. The unary term gives cues on the local appearance while the pairwise term captures the contextual relations between the parts. We extract the features from each frame in a timeslice and examine different temporal aggregation schemes to generate a descriptor for the whole time-slice. Evaluations on the TAP dataset which is well-suited for time-slice activity analysis demonstrate the effectiveness of our approach for the task of uncertainty analysis in activity prediction.

Background subtraction is often used as the first step in video analysis and smart surveillance applications. However, the issue of inconsistent performance across different scenarios due to a lack of flexibility remains a serious concern. To address this, we propose a novel non-parametric, pixel-level background modeling approach based on word dictionaries that draws from traditional codebooks and sample consensus approaches. In this new approach, the importance of each background sample (or word) is evaluated online based on their recurrence among all local observations. This helps build smaller pixel models that are better suited for long-term foreground detection. Combining these models with a frame-level dictionary and local feedback mechanisms leads us to our proposed background subtraction method, coined “PAWCS.” Experiments on the 2012 and 2014 versions of the ChangeDetection.net data set show that PAWCS outperforms 26 previously tested and published methods in terms of overall F-Measure as well as in most categories taken individually. Our results can be reproduced with a C++ implementation available online.

Recognizing human activities from video data is being leveraged for surveillance and human-computer interaction applications. In this paper, we introduce the problem of time-slice activity recognition which aims to explore human activity at a smaller temporal granularity. Time-slice recognition is able to infer human behaviors from a short temporal window. It has been shown that the temporal slice analysis is helpful for motion characterization and in general for video content representation. These studies motivate us to consider time-slices for activity recognition. We present in Figure 1 an overview of our approach based on timeslice action prediction and contrast it with the conventional approaches which recognize actions based on either the whole video sequence (referred as “holistic” approach) or the first part of it (early recognition). Our time-slice approach studies not only the beginning of the action sequence but generalizes this to any short-term observation anywhere in the video sequence. Another key novelty is in the explicit modeling of the uncertainty occurring when predicting actions based on time-slices. TAP Dataset: We introduce a new dataset, named Time-slice Action Prediction (TAP) dataset, to evaluate our proposed feature descriptors and enable future research on this topic. The dataset was created by extracting time-slices from existing public human action datasets (UT-Interaction, HMDB, TV Interaction, and Hollywood datasets) and perform a perception study with multiple annotators giving continuous ratings for each action. The continuous ratings allow to represent the uncertainty in timeslice action prediction. 3 annotators rated each time-slice on how likely a specific action is occurring. For each time-slice and for each action, the annotator was asked to pick one of 5 likelihoods from “Definitely Not Occurring” to “Definitely Occurring”. Figure 3 illustrates how annotators rated for two example videos. Methodology: Stage 1Discriminative segments: When analyzing an interaction, we can definitely recognize the ongoing activity from specific time slices such as “two people are shaking each other’s hands” slice in handshaking activity. To extract discriminative segments from our dataset, we used Fleiss’ kappa coefficient k [2] to measure the reliability of agreement between annotators. For each interaction video, time-slices where the annotators are in complete agreement, i.e. k=1, on definitely including the interaction of interest, are selected as discriminative segments. Stage 2Predict-STIP: Existing STIP detectors are vulnerable to model the inherent uncertainty in partially observed action recognition Figure 2: Human annotation: This figure shows the average rate of 3 annotators for two video examples: hug and push. The label provided by one annotator is converted to a number on a linear scale from 0 to 1 called the average rate. This average rate will be used to evaluate the performance of our method. Time-slices between dashed lines is the discriminative segment of the interaction.

Tracking with a Pan-Tilt-Zoom (PTZ) camera has been a research topic in computer vision for many years. However, it is difficult to assess the progress that has been made because there is no standard evaluation methodology. The difficulty in evaluating PTZ tracking algorithms arises from their dynamic nature. In contrast to other forms of tracking, PTZ tracking involves both locating the target in the image and controlling the motors of the camera to aim it so that the target stays in its field of view. This type of tracking can only be performed online. In this paper, we propose a new evaluation framework based on a virtual PTZ camera. With this framework, tracking scenarios do not change for each experiment and we are able to replicate the main principles of online PTZ camera control and behavior including camera positioning delays, tracker processing delays, and numerical zoom. We tested our evaluation framework with the Camshift tracker to show its viability and to establish baseline results.

Although there has long been interest in foreground background segmentation based on change detection for video surveillance applications, the issue of inconsistent performance across different scenarios remains a serious concern. To address this, we propose a new type of word based approach that regulates its own internal parameters using feedback mechanisms to withstand difficult conditions while keeping sensitivity intact in regular situations. Coined "PAWCS", this method's key advantages lie in its highly persistent and robust dictionary model based on color and local binary features as well as its ability to automatically adjust pixel-level segmentation behavior. Experiments using the 2012 Change Detection.net dataset show that it outranks numerous recently proposed solutions in terms of overall performance as well as in each category. A complete C++ implementation based on OpenCV is available online.

Foreground/background segmentation via change detection in video sequences is often used as a stepping stone in high-level analytics and applications. Despite the wide variety of methods that have been proposed for this problem, none has been able to fully address the complex nature of dynamic scenes in real surveillance tasks. In this paper, we present a universal pixel-level segmentation method that relies on spatiotemporal binary features as well as color information to detect changes. This allows camouflaged foreground objects to be detected more easily while most illumination variations are ignored. Besides, instead of using manually set, frame-wide constants to dictate model sensitivity and adaptation speed, we use pixel-level feedback loops to dynamically adjust our method's internal parameters without user intervention. These adjustments are based on the continuous monitoring of model fidelity and local segmentation noise levels. This new approach enables us to outperform all 32 previously tested state-of-the-art methods on the 2012 and 2014 versions of the ChangeDetection.net dataset in terms of overall F-Measure. The use of local binary image descriptors for pixel-level modeling also facilitates high-speed parallel implementations: our own version, which used no low-level or architecture-specific instruction, reached real-time processing speed on a midlevel desktop CPU. A complete C++ implementation based on OpenCV is available online.

This paper proposes a method to obtain fronto-parallel (side-view) body part trajectories for a walk observed from an arbitrary view. The method is based on homography transformations computed for each gait half-cycle detected in the walk. Each homography maps the body part trajectories to a simulated side view of the walk. The proposed method is stable as the resulting normalized trajectories are not influenced by missing or omitted parts of the raw trajectories. Experiments confirm that normalized trajectories are in agreement with the ones that would be obtained from a side view.

Addresses the problem of building generic 3D models of structured objects on the basis of single 2D intensity images. In the context of the paper, generic modeling refers to the situation where analysis of the image information is performed on the sole basis of generic knowledge. That is, no a priori knowledge about the specific quantitative shape properties of the objects of interest is ever assumed. Moreover, images of interest are realistic. For instance, they may contain complex foreground 3D objects with textures and shadows, and a cluttered background. Objects are modeled by their constituent parts and connections. Therefore, a partly occluded object could be recognized from its model. Part models are based on geons, which are a set of qualitative generalized cylinders. An overview of the architecture of the modeling system is presented, along with the functionality of each subsystem and processing results.

Proposes an approach to finding minimal descriptions of planar curves (open or closed) as sets of constant curvature segments (primitives such as straight line segments and/or circular arcs). The algorithm proceeds according to a multiscale segmentation and approximation of curves, and an intra- and inter-scale classification of this overall process. Its ultimate goal is to find a set of adequate pairs composed of one scale and one or several set(s) of constant curvature segments to best describe the shape of a curve.

This paper proposes a new approach for the multiscale segmentation and approximation of 2D contours. Its ultimate goal is to find the best set of constant curvature segments (straight line segments and/or circular arcs) to describe the contour in a way that respects its actual shape for recognition purposes. This approach is strictly based on discrete geometry principles, and the resulting algorithm named MuscaGrip (multiscale segmentation and contour approximation based on the geometry of regular inscribed polygons) computes, at multiple scales, two grouping processes.

We present an algorithm that reduces significantly the level of the registration errors between all pairs in a set of range views. This algorithm refines initial estimates of the transformation matrices obtained from either the calibrated acquisition setup or a crude manual alignment. It is an instance of a category of registration algorithms known as iterated closest-point (ICP) algorithms. The algorithm considers the network of views as a whole and minimizes the registration errors of all views simultaneously. This leads to a well-balanced network of views in which the registration errors are equally distributed, an objective not met by previously published ICP algorithms which all process the views sequentially. Experimental results show that this refinement technique improves the calibrated registrations and the quality of the integrated model for complex multi-part objects. In the case of scenes comprising man-made objects of very simple shapes, the basic algorithm faces problems common to all ICP algorithms and so must be extended.

Building integrated models of existing 3-D objects is a key requirement for both reverse engineering and object recognition systems. An automatic 3-D model builder goes through three main steps: i) surface sampling from many views, ii) registration of the sampled views, and iii) integration of the registered views. The accuracy obtained depends on the acquisition and registration errors. The latter is critical since a misalignment of the range views causes their noise distributions to be centered around different means, which makes it difficult to reduce the effect of the acquisition error by simple averaging. In this paper, we propose a general algorithm that reduces significantly the level of the registration errors between all pairs in a set of range views. This algorithm refines initial estimates of the transformation matrices obtained from the calibrated acquisition setup. It considers the network of views as a whole and minimizes the registration errors of all views simultaneously. This leads to a well-balanced network of views in which the registration errors are equally distributed. Experimental results show an improvement of both the calibrated registrations and integrated models.>

Many tasks, such as pose determination, object recognition and model building rely on a geometrical description of the visible surfaces derived from scene measurements. Even though much effort has been invested in surface description, robust extraction of surface parameters from scattered and noisy 3-D measurements is still a delicate task. For a given visible surface section, the extracted parameters should not depend on the sensor position in the scene nor on a particular measurement set on this surface. In this work, we show that a viewpoint invariant and stable local surface description can be extracted on sections where measurement constraints are redundant with respect to a polynomial model. A segmentation approach is developed to identify such stable sections. The approach is based on a measurement error model which takes into account the sensor’s viewpoint. An application of the approach to the extraction of straight line sections from single scan 3-D surface profiles is presented. The extracted stable linear sections are stored in a list that includes the estimated descriptive parameters for each section and indices of reliability for each description. The descriptive parameters obtained from images of the objects is compared with pre-measured parameters.

Primal access recognition of visual objects (PARVO), a computer vision system that addresses the problem of fast and generic recognition of unexpected 3D objects from single 2D views, is considered. Recently, recognition by components (RBC), which is a new human image understanding theory, based on some psychological results, has been proposed as an explanation of how PARVO works. However, no systematic computational evaluation of its many aspects has yet been reported. The PARVO system discussed is a first step toward this goal, since its design respects and makes explicit the main assumptions of the proposed theory. It analyzes single-view 2D line drawings of 3D objects typical of the ones used in human image understanding studies. It is designed to handle partially occluded objects of different shape and dimension in various spatial orientations and locations in the image plane. The system is shown to successfully compute generic descriptions and then recognize many common man-made objects.>

Presents a method to compute the inter-frame transformation between two range image views of complex multi-part objects. No exact feature matching is attempted and no initial approximate transformation is provided. The method is naturally decomposed into two stages of initial estimation and final refinement of the transformation. A hierarchical triangulation-based surface representation provides an efficient way to select the control points at which the alignment of the two surfaces is to be evaluated. This representation also permits the selection of a manageable number of initial transformations among which at least one is to be in the parametric neighborhood of the actual transformation. Experimental results show that the computed transformation between two views of a complex multi-part object may provide angles of rotation within a fraction of a degree of the actual ones.>

The authors describe the geometrical criteria which define viewpoint-invariant features to be extracted from 2-D line drawings of 3-D objects. They also discuss the extraction of these features, which forms the initial stage of a generic object recognition system, the Primal Access Recognition of Visual Objects (PARVO) system. In this system, part-based qualitative descriptions are built and matched to coarse 3-D object models for recognition. The segmentation and labeling of the constituent parts of an object rely on the 3-D properties inferred from the presence of its 2-D features. The original motivation for PARVO its recognition by components, a theory of human image understanding from the field of psychology. Definitions of the geometrical criteria defining the viewpoint-invariant features are introduced. Examples of results obtained by applying these criteria to a typical line drawing are shown.>