Motorcycles are Vulnerable Road Users (VRU) and as such, in addition to bicycles and pedestrians, they are the traffic actors most affected by accidents in urban areas. Automatic video processing for urban surveillance cameras has the potential to effectively detect and track these road users. The present review focuses on algorithms used for detection and tracking of motorcycles, using the surveillance infrastructure provided by CCTV cameras. Given the importance of results achieved by Deep Learning theory in the field of computer vision, the use of such techniques for detection and tracking of motorcycles is also reviewed. The paper ends by describing the performance measures generally used, publicly available datasets (introducing the Urban Motorbike Dataset (UMD) with quantitative evaluation results for different detectors), discussing the challenges ahead and presenting a set of conclusions with proposed future work in this evolving area.

Recognising human actions in untrimmed videos is an important challenging task. An effective three‐dimensional (3D) motion representation and a powerful learning model are two key factors influencing recognition performance. In this study, the authors introduce a new skeleton‐based representation for 3D action recognition in videos. The key idea of the proposed representation is to transform 3D joint coordinates of the human body carried in skeleton sequences into RGB images via a colour encoding process. By normalising the 3D joint coordinates and dividing each skeleton frame into five parts, where the joints are concatenated according to the order of their physical connections, the colour‐coded representation is able to represent spatio‐temporal evolutions of complex 3D motions, independently of the length of each sequence. They then design and train different deep convolutional neural networks based on the residual network architecture on the obtained image‐based representations to learn 3D motion features and classify them into classes. Their proposed method is evaluated on two widely used action recognition benchmarks: MSR Action3D and NTU‐RGB+D, a very large‐scale dataset for 3D human action recognition. The experimental results demonstrate that the proposed method outperforms previous state‐of‐the‐art approaches while requiring less computation for training and prediction.

Detecting human actions in long untrimmed videos is a challenging problem. Existing temporal-action detection methods have difficulties in finding the precise starting and ending times of the actions in untrimmed videos. In this letter, we propose a temporal-action detection framework that can detect multiple actions in an end-to-end manner, based on a Bag of Discriminant Snippets (BoDS). BoDS is based on the observation that multiple actions and the background classes have similar snippets, which cause incorrect classification of action regions and imprecise boundaries. We solve this issue by finding the key-snippets from the training data of each class and compute their discriminative power, which is used in BoDS encoding. During testing of an untrimmed video, we find the BoDS representation for multiple candidate proposals and find their class label based on a majority voting scheme. We test BoDS on the Thumos14 and ActivityNet datasets and obtain state-of-the-art results. For the sports subset of ActivityNet dataset, we obtain a mean Average Precision (mAP) value of 29% at 0.7 temporal Intersection over Union (tIoU) threshold. For the Thumos14 dataset, we obtain a significant gain in terms of mAP, i.e., improving from 20.8% to 31.6% at tIoU = 0.7.

Given the high demand for automated systems for human action recognition, great efforts have been undertaken in recent decades to progress the field. In this paper, we present frameworks for single and multi-viewpoints action recognition based on Space-Time Volume (STV) of human silhouettes and 3D-Histogram of Oriented Gradient (3D-HOG) embedding. We exploit fast-computational approaches involving Principal Component Analysis (PCA) over the local feature spaces for compactly describing actions as combinations of local gestures and L2-Regularized Logistic Regression (L2-RLR) for learning the action model from local features. Outperforming results on Weizmann and i3DPost datasets confirm efficacy of the proposed approaches as compared to the baseline method and other works, in terms of accuracy and robustness to appearance changes.

In this paper, we propose a novel encoding method for the representation of human action videos, that we call Discriminative Action Vector of Locally Aggregated Descriptors (DA-VLAD). DA-VLAD is motivated by the fact that there are many unnecessary and overlapping frames that cause non-discriminative codewords during the training process. DA-VLAD deals with this issue by extracting class-specific clusters and learning the discriminative power of these codewords in the form of informative weights. We use these discriminative action weights with standard VLAD encoding as a contribution of each codeword. DA-VLAD reduces the inter-class similarity efficiently by diminishing the effect of common codewords among multiple action classes during the encoding process. We present the effectiveness of DA-VLAD on two challenging action recognition datasets: UCF101 and HMDB51, improving the state-of-the-art with accuracies of 95.1 % and 80.1 % respectively.

We propose a novel skeleton-based representation for 3D action recognition in videos using Deep Convolutional Neural Networks (D-CNNs). Two key issues have been addressed: First, how to construct a robust representation that easily captures the spatial-temporal evolutions of motions from skeleton sequences. Second, how to design D-CNNs capable of learning discriminative features from the new representation in a effective manner. To address these tasks, a skeleton-based representation, namely, SPMF (Skeleton Pose-Motion Feature) is proposed. The SPMFs are built from two of the most important properties of a human action: postures and their motions. Therefore, they are able to effectively represent complex actions. For learning and recognition tasks, we design and optimize new D-CNNs based on the idea of Inception Residual networks to predict actions from SPMFs. Our method is evaluated on two challenging datasets including MSR Action3D and NTU-RGB+D. Experimental results indicated that the proposed method surpasses state-of-the-art methods whilst requiring less computation.

This letter proposes a method for the generation of temporal action proposals for the segmentation of long uncut video sequences. The presence of consecutive multiple actions in video sequences makes the temporal segmentation a challenging problem due to the unconstrained nature of actions in space and time. To address this issue, we exploit the nonaction segments present between the actual human actions in uncut videos. From the long uncut video, we compute the energy of consecutive nonoverlapping motion history images (MHIs), which provides spatiotemporal information of motion. Our proposals from MHIs (PMHI) are based on clustering the MHIs into actions and nonaction segments by detecting minima from the energy of MHIs. PMHI efficiently segments the long uncut videos into a small number of nonoverlapping temporal action proposals. The strength of PMHI is that it is unsupervised, which alleviates the requirement for any training data. Our temporal action proposal method outperforms the existing proposal methods on the Multi-view Human Action video (MuHAVi)-uncut and Computer Vision and Pattern recognition (CVPR) 2012 Change Detection datasets with an average recall rate of 86.1% and 86.0%, respectively.

In this study, a new multi‐view human action recognition approach is proposed by exploiting low‐dimensional motion information of actions. Before feature extraction, pre‐processing steps are performed to remove noise from silhouettes, incurred due to imperfect, but realistic segmentation. Two‐dimensional motion templates based on motion history image (MHI) are computed for each view/action video. Histograms of oriented gradients (HOGs) are used as an efficient description of the MHIs which are classified using nearest neighbor (NN) classifier. As compared with existing approaches, the proposed method has three advantages: (i) does not require a fixed number of cameras setup during training and testing stages hence missing camera‐views can be tolerated, (ii) requires less memory and bandwidth requirements and hence (iii) is computationally efficient which makes it suitable for real‐time action recognition. As far as the authors know, this is the first report of results on the MuHAVi‐uncut dataset having a large number of action categories and a large set of camera‐views with noisy silhouettes which can be used by future workers as a baseline to improve on. Experimentation results on multi‐view with this dataset gives a high‐accuracy rate of 95.4% using leave‐one‐sequence‐out cross‐validation technique and compares well to similar state‐of‐the‐art approaches.

This study contributes in two ways to the research of multi‐camera object tracking in the context of visual surveillance. Firstly, a semi‐automatic scene calibration method is proposed to deal with mapping a network of cameras with overlapped fields of view onto a single ground plane view, even when the overlap is not substantial. The proposed method uses a semi‐supervised approach that combines tracked blobs with user‐selected line scene features to recover the homographies between camera views that are both simple and accurate. Then, based on the scene calibration information, the intersection points of the projected vertical axis of single camera blobs are used to make object correspondences across multiple views. The method works in mixed environments of both pedestrians and vehicles, and is shown to be accurate and robust against segmentation noise and occlusions. Finally, the advantage of the proposed method is demonstrated by quantitative tracking performance evaluation and comparison against previous methods.

Many man‐made environments are constructed with multiple levels where people walk, joined by stairs, ramps and overpasses. This study proposes a novel method to learn the geometry of a scene containing more than a single ground plane by tracking pedestrians and combining information from multiple views. The method estimates a scene model with multiple planes by measuring the variation of pedestrian heights across each camera's field of view. It segments the image into separate plane regions, estimating the relative depth and altitude for each image pixel, thus building a three‐dimensional reconstruction of the scene. By estimating the multiple planes, the method enables tracking algorithms to follow objects (pedestrians and/or vehicles) that are moving on different ground planes in the scene. The authors also introduce what they believe is the first public dataset with pedestrian traffic on multiple planes to encourage other researchers to compare their work in this field.

This paper addresses the problem of silhouette-based human action segmentation and recognition in monocular sequences. Motion History Images (MHIs), used as 2D templates, capture motion information by encoding where and when motion occurred in the images. Inspired by codebook approaches for object and scene categorization, we first construct a codebook of temporal motion templates by clustering all the MHIs of each particular action. These MHIs capture different actors, speeds and a wide range of camera viewpoints. In this paper, we use a Kohonen's Self-Organizing Map (SOM) to simultaneously cluster the MHI templates and represent them in lower dimensional subspaces. To cope with temporal segmentation, and concurrently carry out action recognition, a new architecture is proposed where the obsrvation MHIs are projected onto all these action-specific manifolds and the Euclidean distance between each MHI and the nearest cluster within each action-manifold constitutes the observation vector of a Markov Model. To estimate the state/action at each time step, we introduce a new method based on Observable Markov Models (OMMs) where the Markov model is augmented with a neutral state. The combination of our action-specific manifolds with the augmented OMM allows to automatically segment and recognize long sequences of consecutive actions, without any prior knowledge about initial and ending frames of each action. Importantly, our method allows to interpolate betweeen training viewpoint and recognizes actions, independently of the camera viewpoint, even from unseen viewpoints.

A novel embedding-based dimensionality reduction approach, called structural Laplacian Eigenmaps, is proposed to learn models representing any concept that can be defined by a set of multivariate sequences. This approach relies on the expression of the intrinsic structure of the multivariate sequences in the form of structural constraints, which are imposed on dimensionality reduction process to generate a compact and data-driven manifold in a low dimensional space. This manifold is a mathematical representation of the intrinsic nature of the concept of interest regardless of the stylistic variability found in its instances. In addition, this approach is extended to model jointly several related concepts within a unified representation creating a continuous space between concept manifolds. Since a generated manifold encodes the unique characteristic of the concept of interest, it can be employed for classification of unknown instances of concepts. Exhaustive experimental evaluation on different datasets confirms the superiority of the proposed methodology to other state-of-the-art dimensionality reduction methods. Finally, the practical value of this novel dimensionality reduction method is demonstrated in three challenging computer vision applications, i.e., view-dependent and view-independent action recognition as well as human-human interaction classification.

Metric learning methods, for person re-identification, estimate a scaling for distances in a vector space that is optimized for picking out observations of the same individual. This paper presents a novel approach to the pedestrian re-identification problem that uses metric learning to improve the state-of-the-art performance on standard public datasets. Very high dimensional features are extracted from the source color image. A first processing stage performs unsupervised PCA dimensionality reduction, constrained to maintain the redundancy in color-space representation. A second stage further reduces the dimensionality, using a Local Fisher Discriminant Analysis defined by a training set. A regularization step is introduced to avoid singular matrices during this stage. The experiments conducted on three publicly available datasets confirm that the proposed method outperforms the state-of-the-art performance, including all other known metric learning methods. Further-more, the method is an effective way to process observations comprising multiple shots, and is non-iterative: the computation times are relatively modest. Finally, a novel statistic is derived to characterize the Match Characteristic: the normalized entropy reduction can be used to define the 'Proportion of Uncertainty Removed' (PUR). This measure is invariant to test set size and provides an intuitive indication of performance.

This paper presents the conceptual and architectural design of a multilayered framework aiming to provide a two handed gesture based visual interactive 3D object oriented environment for software development. We argue that this is a viable, intuitive and attractive approach for software development facilitating natural human computer interaction, thus supporting tasks related to software development in a more effective and efficient way. We illustrate the value of the proposed environment through a data analysis from early evaluations of one prototype system in use. The results and implications of this study are useful for designing effective and efficient gesture based 3D interactive software development environments.

A new generation of advanced surveillance systems is being conceived as a collection of multisensor components such as video, audio, and mobile robots interacting in a cooperating manner to enhance situation awareness capabilities to assist surveillance personnel. The prominent issues that these systems face are the improvement of existing intelligent video surveillance systems, the inclusion of wireless networks, the use of low power sensors, the design architecture, the communication between different components, the fusion of data emerging from different type of sensors, the location of personnel (providers and consumers), and the scalability of the system. This paper focuses on the aspects pertaining to real-time distributed architecture and scalability. For example, to meet real-time requirements, these systems need to process data streams in concurrent environments, designed by taking into account scheduling and synchronization. This paper proposes a framework for the design of visual surveillance systems based on components derived from the principles of real-time networks/data-oriented requirements implementation scheme. It also proposes the implementation of these components using the well-known middleware technology common object request broker architecture. Results using this architecture for video surveillance are presented through an implemented prototype.

Automatic video analysis from urban surveillance cameras is a fast-emerging field based on computer vision techniques. We present here a comprehensive review of the state-of-the-art computer vision for traffic video with a critical analysis and an outlook to future research directions. This field is of increasing relevance for intelligent transport systems (ITSs). The decreasing hardware cost and, therefore, the increasing deployment of cameras have opened a wide application field for video analytics. Several monitoring objectives such as congestion, traffic rule violation, and vehicle interaction can be targeted using cameras that were typically originally installed for human operators. Systems for the detection and classification of vehicles on highways have successfully been using classical visual surveillance techniques such as background estimation and motion tracking for some time. The urban domain is more challenging with respect to traffic density, lower camera angles that lead to a high degree of occlusion, and the variety of road users. Methods from object categorization and 3-D modeling have inspired more advanced techniques to tackle these challenges. There is no commonly used data set or benchmark challenge, which makes the direct comparison of the proposed algorithms difficult. In addition, evaluation under challenging weather conditions (e.g., rain, fog, and darkness) would be desirable but is rarely performed. Future work should be directed toward robust combined detectors and classifiers for all road users, with a focus on realistic conditions during evaluation.

This paper describes a body of multicamera human action video data with manually annotated silhouette data that has been generated for the purpose of evaluating silhouette-based human action recognition methods. It provides a realistic challenge to both the segmentation and human action recognition communities and can act as a benchmark to objectively compare proposed algorithms. The public multi-camera, multi-action dataset is an improvement over existing datasets (e.g. PETS, CAVIAR, soccerdataset) that have not been developed specifically for human action recognition and complements other action recognition datasets (KTH, Weizmann, IXMAS, HumanEva, CMU Motion). It consists of 17 action classes, 14 actors and 8 cameras. Each actor performs an action several times in the action zone. The paper describes the dataset and illustrates a possible approach to algorithm evaluation using a previously published action simple recognition method. In addition to showing an evaluation methodology, these results establish a baseline for other researchers to improve upon.

This paper addresses the problem of silhouette-based human action modeling and recognition, specially when the number of action samples is scarce. The first step of the proposed system is the 2D modeling of human actions based on motion templates, by means of motion history images (MHI). These templates are projected into a new subspace using the Kohonen self organizing feature map (SOM), which groups viewpoint (spatial) and movement (temporal) in a principal manifold, and models the high dimensional space of static templates.The next step is based on the hidden Markov models (HMM) in order to track the map behavior on the temporal sequences of MHI. Every new MHI pattern is compared with the features map obtained during the training. The index of the winner neuron is considered as discrete observation for the HMM. If the number of samples is not enough, a sampling technique, the sampling importance resampling (SIR) algorithm, is applied in order to increase the number of observations for the HMM. Finally, temporal pattern recognition is accomplished by a maximum likelihood (ML) classifier. We demonstrate this approach on two publicly available dataset: one based on real actors and another one based on virtual actors.

This paper proposes and demonstrates a novel method for the detection and classification of individual vehicles and pedestrians in urban scenes. In this scenario, shadows, lights and various occlusions compromise the accuracy of foreground segmentation and hence there are challenges with conventional silhouette-based methods. 2D features derived from histograms of oriented gradients (HOG) have been shown to be effective for detecting pedestrians and other objects. However, the appearance of vehicles varies substantially with the viewing angle and local features may be often occluded. In this paper, a novel method is proposed that overcomes limitations in the use of 2D HOG. Full 3D models are used for the object categories to be detected and the feature patches are defined over these models. A calibrated camera allows an affine transform of the observation into a normalised representation from which ‘3DHOG ’ features are defined. A variable set of interest points is used in the detection and classification processes, depending on which points in the 3D model are visible. Experiments on real CCTV data of urban scenes demonstrate the proposed method. The 3DHOG feature is compared with features based on FFT and simple histograms. A baseline method using overlap between wire-frame models and motion silhouettes is also included. The results demonstrate that the proposed method achieves comparable performance. In particular, an advantage of the proposed method is that it is more robust than motion silhouettes which are often compromised in real data by variable lighting, camera quality and occlusions from other objects. 1.

Video-based methods such as tracking and appearance recognition can be used in the public transport domain to re-acquire an estimate of the whereabouts of a passenger of interest. Starting with the end user requirements, it is argued that the most appropriate performance metric with which to evaluate these methods, is to measure the information gained through their use. This is equivalent to the reduction in uncertainty about the passenger's whereabouts, after tracking and appearance-based measurements are used to update the estimates of which detected persons are likely to be the passenger of interest. The prior estimates are significantly conditioned on the statistics for the likely distributions for entry and traversal through the network: a simple model is described for this purpose. The suitability of tracking and appearance based matching methods are discussed in this context.

We propose a new method for human action recognition from video streams that is fast and robust to noise and to large changes in camera views. We start by extracting features in the Fourier domain once we obtain the bounding boxes containing the silhouettes of a human for a number of video frames representing a basic action. After preprocessing, we divide each space-time volume into space-time sub-volumes (STSV) and compute their corresponding mean-power spectra as our feature vectors. Our features result in high classification performance even with simple distance measures. We perform an experimental comparison, using the same data, between our method and two state-of-the-art methods.

This paper builds on previous work to propose a meta-data standard for video surveillance. The motivation is to promote interoperability. The starting point is the set of requirements under consideration for a Multimedia Application Format. These requirements cover a description of the surveillance system and of the activity in the scene. In addition to this set, appropriate descriptions for the relation between camera and scene are also considered. To improve interoperability between systems and between components of a system, two types of restrictions are proposed. The first proposal is a restricted subset of the MPEG-7 elements that are applicable to the surveillance domain. The second proposal is to use the MPEG-7 tools to include domain-specific taxonomies to restrict the names of elements used in the semantic descriptions. Both proposals are incorporated into examples which demonstrate the use of the standard.

In this paper, a system for automatic robust video surveillance is described and in particular its application to the problem of locating vehicles that stop in prohibited area is discussed. The structure of software for video processing (alarm generation, interface with the operator and information storage) is outlined together with the hardware (Trimedia DSP boards and industrial computers) which constitutes an industrial-grade product. The emphasis on this paper is to demonstrate robust detection and hence we show the results of a performance evaluation process carried out with the UK's i-LIDS "Parked Vehicle " reference dataset.

This paper presents a real time surveillance system that can be deployed within public transport environments. The system is a fully customisable application that can be used to address the requirements for this application domain. The system has been under trial on a number of metropolitan railway systems in the UK and Italy. In this paper we discuss how some of the algorithms employed within the system can be adapted for use with the CREDS datasets.

In video surveillance projects, automatic and real-time event detection solutions are required to guarantee an efficient and cost-effective use of the infrastructure. Many solutions have been proposed to automatically detect a variety of events of interest. However, not all solutions and technologies may satisfy all the requirements of the surveillance scenario. For this reason, performance evaluation of existing event detection solutions becomes an important step in the deployment of video surveillance projects. In this paper, we propose a practical approach that aims at minimizing the ground truth generation problem and the expertise required to evaluate and compare the results by introducing specific requirements of specific event detection scenarios. This approach is believed to be applicable for an initial evaluation of candidate solutions to a specific surveillance scenario before more exhaustive tests in an integrated environment. The proposed method is under evaluation in the framework of the challenge of real-time event detection solutions (CREDS).

On-line surveillance to improve safety and security is a major requirement for the management of public transport networks and other public places. The surveillance task is a complex one involving people, management procedures, and technology. This work describes an architecture that takes into account the distributed nature of the detection processes and the need to allow for different types of devices and actuators. This was part of a major European initiative on intelligent transport systems. Because of the dominant nature of closed circuit television in surveillance, This work describes in detail a computer-vision module used in the system and its particular ability to detect situations of interest in busy conditions. The system components have been implemented, integrated, and tested in real metropolitan railway environments and are considered to be the first step toward providing ambient intelligence in such complex scenarios. Results are presented that not only deal with detection performance, but also on the perception of people who used the system on its effectiveness and potential impact.

Using a simple but robust real-time algorithm that allows tracking of multiple objects in complex environments, the authors propose a wide range application in automatic surveillance by linking blobs' information, in terms of trajectories and positions, with the events to be detected. Real-time is achieved reducing the computation time avoiding the use of colour in the segmentation stage and statistical prediction in the tracking stage and using instead segmentation based on luminance contrast and tracking based on the overlapping information from bounding boxes from two consecutive frames.

The estimation of the number of people in an area under surveillance is very important for the problem of crowd monitoring. When an area reaches an occupation level greater than the projected one, people's safety can be in danger. This paper describes a new technique for crowd density estimation based on Minkowski fractal dimension. The fractal dimension has been widely used to characterize data texture in a large number of physical and biological sciences. The results of our experiments show that fractal dimension can also be used to characterize levels of people congestion in images of crowds. The proposed technique is compared with a statistical and a spectral technique, in a test study of nearly 300 images of a specific area of the Liverpool Street Railway Station, London, UK. Results obtained in this test study are presented.

This paper presents a technique for oriented texture classification which is based on the Hough transform and Kohonen's (1990) neural network model. In this technique, oriented texture features are extracted from the Hough space by means of two distinct strategies. While the first operates on a non-uniformly sampled Hough space, the second concentrates on the peaks produced in the Hough space. The described technique gives good results for the classification of oriented textures, a common phenomenon in nature underlying an important class of images. Experimental results are presented to demonstrate the performance of the new technique in comparison with an implemented technique based on Gabor filters.

This paper introduces the data oriented requirements implementation scheme (DORIS) design strategy and the data interaction architecture demonstrator (DIADEM), by describing their use for a multiple-robot workspace simulation. The design process, the hardware and the software for the DIADEM are outlined, followed by a discussion of the design used for the simulation.>

The Hough Transform (HT) is an efficient method to extract geometric features from an image which works fairly well for images that contain noise and occlusion. However, its performance decreases with image and parameter space quantisation noise. Accuracy is limited by memory costs and oversampling effects. This paper describes two HT variants based on analytical least squares refinement based on a conventional HT voting stage. Both methods reduce memory requirements. A number of experimental results are presented and compared with the standard HT.>

This paper presents a new Hough function based on a Mahalanobis distance measure that incorporates a formal stochastic model for measurement and model noise. Thus, the effects of image and parameter space quantisation can be incorporated directly. An extended Kalman filtering process is proposed combined with this flexible model for feature refinement, which is particularly applicable when using coarse parameter resolutions. For a given resolution, the method provides better performance than the SHT.>

In adaptive robots on-line sensor corrections are generally specified relative to a world co-ordinate system, whereas the manipulator is actuated in a joint co-ordinate system. Thus the real-time execution of what is known as the Inverse Kinematics Algorithm (IKA) is very important. To decrease execution times, parallel processing supported by a multi-processor architecture were used. Of special interest was the use of the Ada programming language, normally associated with large systems, and its inherent facilities for parallel processing. Taking as an example an inverse kinematics algorithm for the RTX robot, a method was used to change the algorithm from a sequential form to a parallel form suitable for implementation on a multi-processor system. The Algorithm was then implemented on Ada targeted to a multi-transputer system. Performance was then evaluated for a variety of transputer configurations and compared to equivalent implementations written in Occam. We conclude that although there are a number of transputer configurations that can fulfil realistic performance criteria, the selected implementation of Ada does not make it a viable option for a cost-effective implementation of co-ordinate transformation for the selected robot.