Avoiding bottleneck situations in crowds is critical for the safety and comfort of people at large events or in public transportation. Based on the work of Lagrangian motion analysis we propose a novel video-based bottleneck-detector by identifying characteristic stowage patterns in crowd-movements captured by optical flow fields. The Lagrangian framework allows to assess complex time-dependent crowd-motion dynamics at large temporal scales near the bottleneck by two dimensional Lagrangian fields. In particular we propose long-term temporal filtered Finite Time Lyapunov Exponents (FTLE) fields that provide towards a more global segmentation of the crowd movements and allows to capture its deformations when a crowd is passing a bottleneck. Finally, these deformations are used for an automatic spatio-temporal detection of such situations. The performance of the proposed approach is shown in extensive evaluations on the existing Jülich and AGO-RASET datasets, that we have updated with ground truth data for spatio-temporal bottleneck analysis.

Today's multi-object tracking approaches benefit greatly from nearly perfect object detections when following the popular tracking-by-detection scheme. This allows for extremely simple but accurate tracking methods which completely rely on the input detections as the high-speed IOU tracker. For real world applications, few missing detections cause a high number of ID switches and fragmentations which degrades the quality of the tracks significantly. We show that this problem can be efficiently overcome if the tracker falls back to visual single-object tracking in cases where no object detection is available. In several experiments we show for different visual trackers that the number of ID switches and fragmentations can be reduced by a large amount while maintaining high tracking speeds and outperforming the state-of-the art for the UA-DETRAC and VisDrone datasets.

A desirable smart traffic-monitoring and street-safety system can elicit and support the intervention of law enforcement agencies or medical staff. Recently, there has been a dramatically higher demand for such smart systems. To this end, the International Workshop on Traffic and Street Surveillance for Safety and Security (IWT4S) was organized in conjunction with the 15th IEEE International Conference on Advanced Video and Signal-based Surveillance (AVSS 2018). Our goal is to advance the state-of-the-art detection and tracking algorithms and provide a comprehensive performance evaluation for them. We evaluate 5 submitted detection and 7 submitted tracking methods on the large-scale UA-DETRAC benchmark, and the results are shared publicly on the website http://detrac-db. rit.albany.edu. We expect this challenge to advance the research and development of new detection and tracking methods for transportation applications.

The performance of optical flow algorithms greatly depends on the specifics of the content and the application for which it is used. Existing and well established optical flow datasets are limited to rather particular contents from which none is close to crowd behavior analysis; whereas such applications heavily utilize optical flow. We introduce a new optical flow dataset exploiting the possibilities of a recent video engine to generate sequences with ground-truth optical flow for large crowds in different scenarios. We break with the development of the last decade of introducing ever increasing displacements to pose new difficulties. Instead we focus on real-world surveillance scenarios where numerous small, partly independent, non rigidly moving objects observed over a long temporal range pose a challenge. By evaluating different optical flow algorithms, we find that results of established datasets can not be transferred to these new challenges. In exhaustive experiments we are able to provide new insight into optical flow for crowd analysis. Finally, the results have been validated on the real-world UCF crowd tracking benchmark while achieving competitive results compared to more sophisticated state-of-the-art crowd tracking approaches.

In a broad range of computer vision tasks, convolutional neural networks (CNNs) are one of the most prominent techniques due to their outstanding performance. Yet it is not trivial to find the best performing network structure for a specific application because it is often unclear how the network structure relates to the network accuracy. We propose an evolutionary algorithm-based framework to automatically optimize the CNN structure by means of hyper-parameters. Further, we extend our framework towards a joint optimization of a committee of CNNs to leverage specialization and cooperation among the individual networks. Experimental results show a significant improvement over the state-of-the-art on the well-established MNIST dataset for hand-written digits recognition.

Lagrangian theory provides a rich set of tools for analyzing non-local, long-term motion information in computer vision applications. Based on this theory, we present a specialized Lagrangian technique for the automated detection of violent scenes in video footage. We present a novel feature using Lagrangian direction fields that is based on a spatio-temporal model and uses appearance, background motion compensation, and long-term motion information. To ensure appropriate spatial and temporal feature scales, we apply an extended bag-of-words procedure in a late-fusion manner as a classification scheme on a per-video basis. We demonstrate that the temporal scale, captured by the Lagrangian integration time parameter, is crucial for violence detection and show how it correlates to the spatial scale of characteristic events in the scene. The proposed system is validated on multiple public benchmarks and non-public, real-world data from the London Metropolitan Police. Our experiments confirm that the inclusion of Lagrangian measures is a valuable cue for automated violence detection and increases the classification performance considerably compared with the state-of-the-art methods.

Sparse motion estimation with local optical flow methods is fundamental for a wide range of computer vision application. Classical approaches like the pyramidal Lucas-Kanade method (PLK) or more sophisticated approaches like the Robust Local Optical Flow (RLOF) fail when it comes to environments with illumination changes and/or long-range motions. In this work we focus on these limitations and propose a novel local optical flow framework taking into account an illumination model to deal with varying illumination and a prediction step based on a perspective global motion model to deal with long-range motions. Experimental results shows tremendous improvements, e.g. 56% smaller error for dense motion fields on the KITTI and an about 76% smaller error for sparse motion fields on the Sintel dataset.

Optical flow methods integrating sparse point correspondences have made significant contribution in the field of optical flow estimation. Especially for the goal of estimating motion accurately and efficiently, sparse-to-dense interpolation schemes for feature point matches have shown outstanding performances. Concurrently, local optical flow methods have been significantly improved with respect to long-range motion estimation in environments with varying illumination. This motivates us to propose a sparse-to-dense approach based on the Robust Local Optical Flow method. Compared to state-of-the-art methods the proposed approach is significantly faster while retaining competitive accuracy on Middlebury, KITTI 2015 and MPI-Sintel data-set.

In many computer vision applications local optical flow methods are still a widely used. Such methods, like the Pyramidal Lucas Kanade and the Robust Local Optical Flow, have to address the trade-off between run time and accuracy. In this work we propose an extension to these methods that improves the accuracy especially at object boundaries. This extension makes use of the cross based variable support region generation proposed in [1] accounting for local intensity discontinuities. In the evaluation using Middlebury data set we prove the ability of the proposed extension to increase the accuracy by a slight increase of run time.

We propose a new methodology for crowd analysis by introducing the concept of Multi-Person Density. Using a state-of-the-art feature tracking algorithm, representative low-level features and their long-term motion information are extracted and combined into a human detection model. In contrast to previously proposed techniques, the proposed method takes small camera motion into account and is not affected by camera shaking. This increases the robustness of separating crowd features from background and thus opens a whole new field for application of these techniques in non-static CCTV cameras. We show the effectiveness of our approach on various test videos and compare it to state-of-the-art people counting methods.

This article presents a theoretical framework to decrease the computation effort of the Robust Local Optical Flow method which is based on the Lucas Kanade method. We show mathematically, how to transform the iterative scheme of the feature tracker into a system of bilinear equations and thus estimate the motion vectors directly by analyzing its zeros. Furthermore, we show that it is possible to parallelise our approach efficiently on a GPU, thus, outperforming the current OpenCV-OpenCL implementation of the pyramidal Lucas Kanade method in terms of runtime and accuracy. Finally, an evaluation is given for the Middlebury Optical Flow and the KITTI datasets.

The selection of regions or sets of points to track is a key task in motion-based video analysis, which has significant performance effects in terms of accuracy and computational efficiency. Computational efficiency is an unavoidable requirement in video surveillance applications. Well established methods, e.g. Good Features to Track, select points to be tracked based on appearance features such as cornerness and therefore neglecting the motion exhibited by the selected points. In this paper, we propose an interest point selection method that takes into account the motion of previously tracked points in order to constrain the number of point trajectories needed. By defining pair-wise temporal affinities between trajectories and representing them in a minimum spanning tree, we achieve a very efficient clustering. The number of trajectories assigned to each motion cluster is adapted by initializing and removing tracked points by means of feed-back. Compared to the KLT tracker, we save up to 65% of the points to track, therefore gaining in efficiency while not scarifying accuracy.

The availability of dense motion information in computer vision domain allows for the effective application of Lagrangian techniques that have their origin in fluid flow analysis and dynamical systems theory. A well established technique that has been proven to be useful in image-based crowd analysis are Finite Time Lyapunov Exponents (FTLE). Based on this, we present a method to detect people carrying object and describe a methodology how to apply established flow field methods onto the problem of describing individuals. Further, we reinterpret Lagrangian features in relation to the underlying motion process and show their applicability towards the appearance modeling of pedestrians. This definition allows to increase performance of state-of-the-art methods and is shown to be robust under varying parameter settings and different optical flow extraction approaches.

Human action recognition requires the description of complex motion patterns in image sequences. In general, these patterns span varying temporal scales. In this context, Lagrangian methods have proven to be valuable for crowd analysis tasks such as crowd segmentation. In this paper, we show that, besides their potential in describing large scale motion patterns, Lagrangian methods are also well suited to model complex individual human activities over variable time intervals. We use Finite Time Lyapunov Exponents and time-normalized arc length measures in a linear SVM classification scheme. We evaluated our method on the Weizmann and KTH datasets. The results demonstrate that our approach is promising and that human action recognition performance is improved by fusing Lagrangian measures.

The extraction of motion patterns from image sequences based on the optical flow methodology is an important and timely topic among visual multi media applications. In this work we will present a novel framework that combines the optical flow methodology from image processing with methods developed for the Lagrangian analysis of time-dependent vector fields. The Lagrangian approach has been proven to be a valuable and powerful tool to capture the complex dynamic motion behavior within unsteady vector fields. To come up with a compact and applicable framework, this paper will provide concepts on how to compute trajectory-based Lagrangian measures in series of optical flow fields, a set of basic measures to capture the essence of the motion behavior within the image, and a compact hierarchical, feature-based description of the resulting motion features. The resulting framework will bee shown to be suitable for an automated image analysis as well as compact visual analysis of image sequences in its spatio-temporal context. We show its applicability for the task of motion feature description and extraction on different temporal scales, crowd motion analysis, and automated detection of abnormal events within video sequences.

This paper is motivated by the problem of local motion estimation via robust regression with linear models. In order to increase the robustness of the motion estimates, we propose a novel robust local optical flow approach based on a modified Hampel estimator. We show the deficiencies of the least squares estimator used by the standard Kanade-Lucas-Tomasi (KLT) tracker when the assumptions made by Lucas-Kanade are violated. We propose a strategy to adapt the window sizes to cope with the generalized aperture problem. Finally, we evaluate our method on the Middlebury and MIT dataset and show that the algorithm provides excellent feature tracking performance with only slightly increased computational complexity compared to KLT. To facilitate further development, the presented algorithm can be downloaded from http://www.nue.tu-berlin.de/menue/forschung/projekte/rlof.

In this paper we present a decentralized surveillance network composed of IP video cameras, analysis devices and a central node which collects information and displays it in a 3D model of the complete area. The exchange of information between all components in the surveillance network takes place according to the ONVIF specification, therefore ensuring interoperability between products complying with the specification and flexibility regarding the integration of new devices and services. The collected information is displayed in a 3D model of the surveilled area, therefore providing a comfortable overview of the activity in large environments and offering the user an intuitive way to eventually interact with network devices.

In this paper we present an approach for the efficient computation of optical flow fields in real-time and provide implementation details. Proposing a modification of the popular Lucas-Kanade energy functional based on integral projections allows us to speed up the method notably. We show the potential of this method which can compute dense flow fields of 640×480 pixels at a speed of 4 fps in a GPU implementation based on the OpenCL framework. Working on sparse optical flow fields of up to 17,000 points, we reach execution times of 70 fps. Optical flow methods are used in many different areas, the proposed method speeds up current surveillance algorithms used for scene description and crowd analysis or Augmented Reality and robot navigation applications.

Detecting static objects in video sequences has a high relevance in many surveillance scenarios like airports and railwaystations. In this paper we propose a system for the detection of static objects in crowded scenes that, based on the detection of two background models learning at different rates, classifies pixels with the help of a finite-state machine. The background is modelled by two mixtures of Gaussians with identical parameters except for the learning rate. The state machine provides the meaning for the interpretation of the results obtained from background subtraction and can be used to incorporate additional information cues, obtaining thus a flexible system specially suitable for real-life applications. The system was built in our surveillance application and successfully validated with several public datasets.

This paper describes a robust method for the local optical flow estimation and the KLT feature tracking performed on the GPU. Therefore we present an estimator based on the L2norm with robust characteristics. In order to increase the robustness at discontinuities we propose a strategy to adapt the used region size. The GPU implementation of our approach achieves real-time (>;25 fps) performance for High Definition (HD) video sequences while tracking several thousands of points. The benefit of the suggested enhancement is illustrated on the Middlebury optical flow benchmark.

Detecting people carrying objects is a commonly formulated problem as a first step to monitor interactions between people and objects. Recent work relies on a precise foreground object segmentation, which is often difficult to achieve in video surveillance sequences due to a bad contrast of the foreground objects with the scene background, abrupt changing light conditions and small camera vibrations. In order to cope with these difficulties we propose an approach based on motion statistics. Therefore we use a Gaussian mixture motion model (GMMM) and, based on that model, we define a novel speed and direction independent motion descriptor in order to detect carried baggage as those regions not fitting in the motion description model of an average walking person. The system was tested with the public dataset PETS2006 and a more challenging dataset including abrupt lighting changes and bad color contrast and compared with existing systems, showing very promising results.