Abstract This paper investigates a new method to simulate pedestrian crowd movement in a large and complex virtual environment, representing a public space such as a shopping mall. To demonstrate pedestrian dynamics, we consider groups of pedestrians of different size, sharing a crowded environment. A pedestrian has its own characteristics, such as gender, age, position, velocity, and energy. The proposed method uses a multi-group microscopic model to generate real-time trajectories for all people moving in the defined virtual environment. Additionally, a dynamic model is introduced for modelling group behaviour. Based on the proposed method, all pedestrians in each group can continuously adjust their attributes and optimize their path towards the desired visiting targets, while avoiding obstacles and other pedestrians. Simulation results show that the proposed method can describe a realistic simulation of dynamic behaviour.

This paper investigates a new method to simulate pedestrian crowd movement in a large and complex virtual environment, representing a public place such as a shopping mall. To demonstrate pedestrian dynamics, we consider different group sizes with various categories of pedestrian sharing a crowded environment. Each category has its own characteristics, such as gender, age, position, velocity, and energy. The proposed method uses the multi-group microscopic model to generate real-time trajectories for all people moving in the defined virtual environment. Additionally, an agent-based model is introduced for modelling group behaviour. Based on the proposed method, every pedestrian in each group can continuously adjust their attributes and optimize their path towards the desired visiting points, while avoiding obstacles and other pedestrians. The simulation results show that the proposed method can describe a realistic simulation of dynamic behaviour in a given virtual environment.

Abstract In this paper, we investigate the multi‐goal path planning problem to find the shortest collision‐free path connecting a given set of goal points located in the robot working environment. This problem combines two sub‐problems: first, optimize the sequence of the goal points located in the free workspace; second, compute the shortest collision‐free path between goal points. In this study, a genetic algorithm is used to optimize the sequence of the goal points that the robot should visit. Once the sequence of the goal points is available, our developed method called Boundary Node Method (BNM) is applied to generate an initial collision‐free path between every pair of the sequenced goal points. Subsequently, an additional developed method called Path Enhancement Method (PEM) is used to find an optimal or near‐optimal path by reducing the overall initial path length. In the BNM, the robot and its immediate surroundings are simulated by a nine‐node quadrilateral element, where the centroid node represents the robot's position. The robot moves between goals with boundary nodes in the working environment depending on the boundary node's potential value. The potential value of each point in the working environment is calculated based on the proposed potential function. Additionally, this article investigates the multi‐goal path planning problem for multi‐robot systems, when each goal reached by several robots. Finally, simulations and experiments are performed on a real mobile robot to demonstrate the effectiveness of the developed methods.

A novel deep neural network is proposed, for accurate and robust crowd counting. Crowd counting is a complex task, as it strongly depends on the deployed camera characteristics and, above all, the scene perspective. Crowd counting is essential in security applications where Internet of Things (IoT) cameras are deployed to help with crowd management tasks. The complexity of a scene varies greatly, and a medium to large scale security system based on IoT cameras must cater for changes in perspective and how people appear from different vantage points. To address this, our deep architecture extracts multi-scale features with a pyramid contextual module to provide long-range contextual information and enlarge the receptive field. Experiments were run on three major crowd counting datasets, to test our proposed method. Results demonstrate our method supersedes the performance of state-of-the-art methods.

Deep Neural networks (DNN) dominate the state of art results in computer vision (CV) and other fields. One of the primary reasons why DNN outperform existing algorithms is that these produce superior results when more labelled data are used, unlike classic CV techniques. Nonetheless, it is well known that DNN requires a very large amount of data to generalise well. Collecting and labelling these datasets are expensive, time-consuming and sometimes impossible. Therefore, researchers tried to use alternative techniques, such as graphics simulators to automatically generate labelled datasets. However, these techniques are still expensive and require domain knowledge to produce good datasets. In this paper, therefore, a graphics simulator is presented which automatically generates multi-model datasets in real-time providing the corresponding ground truth and annotation. The tool concentrates on pedestrian and crowd analysis including 3D human pose estimation, pedestrian detection as well as crowd density and flow estimation.

In this work, we pose the question whether it is possible to design and train an autoencoder model in an end-to-end fashion to learn representations in the multivariate Bernoulli latent space, and achieve performance comparable with the state-of-the-art variational methods. Moreover, we investigate how to generate novel samples and perform smooth interpolation and attributes modification in the binary latent space. To meet our objective, we propose a simplified, deterministic model with a straight-through gradient estimator to learn the binary latents and show its competitiveness with the latest VAE methods. Furthermore, we propose a novel method based on a random hyperplane rounding for sampling and smooth interpolation in the latent space. Our method performs on a par or better than the current state-of-the-art methods on common CelebA, CIFAR-10 and MNIST datasets.

Within this work a novel semi-supervised learning technique is introduced based on a simple iterative learning cycle together with learned thresholding techniques and an ensemble decision support system. State-of-the-art model performance and increased training data volume are demonstrated, through the use of unlabelled data when training deeply learned classification models. The methods presented work independently from the model architectures or loss functions, making this approach applicable to a wide range of machine learning and classification tasks. Evaluation of the proposed approach is performed on commonly used datasets when evaluating semi-supervised learning techniques as well as a number of more challenging image classification datasets (CIFAR-100 and a 200 class subset of ImageNet).

Deep learning has greatly improved pattern recognition algorithms, however its strength is also its weakness: training data must be plentiful, varied and, above all, annotated. Annotating data for crowd analysis in unrealistic: for instance, consider the task of accurately counting people in a dense crowd. Such task would require hours of painstaking manual marking where people are in a crowd that might have been captured from far field. We propose a method that builds controlled simulations of people moving in a real environment; people are modelled as synthetic humanoids with realistic appearance, while the background is the actual image of a scene. Controlling illumination, dynamics of people and density in the scene, we can virtually generate an infinite number of simulations and very large annotated datasets. This paper demonstrates how the performance of a conventional deep network, used for a crowd analysis task, can be improved by using simulated datasets.

Unmanned Aerial vehicles (UAV) are a promising technology for smart farming related applications. Aerial monitoring of agriculture farms with UAV enables key decision-making pertaining to crop monitoring. Advancements in deep learning techniques have further enhanced the precision and reliability of aerial imagery based analysis. The capabilities to mount various kinds of sensors (RGB, spectral cameras) on UAV allows remote crop analysis applications such as vegetation classification and segmentation, crop counting, yield monitoring and prediction, crop mapping, weed detection, disease and nutrient deficiency detection and others. A significant amount of studies are found in the literature that explores UAV for smart farming applications. In this paper, a review of studies applying deep learning on UAV imagery for smart farming is presented. Based on the application, we have classified these studies into five major groups including: vegetation identification, classification and segmentation, crop counting and yield predictions, crop mapping, weed detection and crop disease and nutrient deficiency detection. An in depth critical analysis of each study is provided.

Smart video sensors for applications related to surveillance and security are IOT-based as they use Internet for various purposes. Such applications include crowd behaviour monitoring and advanced decision support systems operating and transmitting information over internet. The analysis of crowd and pedestrian behaviour is an important task for smart IoT cameras and in particular video processing. In order to provide related behavioural models, simulation and tracking approaches have been considered in the literature. In both cases ground truth is essential to train deep models and provide a meaningful quantitative evaluation. We propose a framework for crowd simulation and automatic data generation and annotation that supports multiple cameras and multiple targets. The proposed approach is based on synthetically generated human agents, augmented frames and compositing techniques combined with path finding and planning methods. A number of popular crowd and pedestrian data sets were used to validate the model, and scenarios related to annotation and simulation were considered.

In this paper we present the design and evaluation of an end-to-end trainable, deep neural network with a visual attention mechanism for memorability estimation in still images. We analyze the suitability of transfer learning of deep models from image classification to the memorability task. Further on we study the impact of the attention mechanism on the memorability estimation and evaluate our network on the SUN Memorability and the LaMem datasets. Our network outperforms the existing state of the art models on both datasets in terms of the Spearman's rank correlation as well as the mean squared error, closely matching human consistency.

In this work, we extract rich representations of crowd behavior from video using a fine-tuned deep convolutional neural residual network. Using spatial partitioning trees we create subclasses within the feature maps from each of the crowd behavior attributes (classes). Features from these subclasses are then regularized using an eigen modeling scheme. This enables to model the variance appearing from the intra-subclass information. Low dimensional discriminative features are extracted after using the total subclass scatter information. Dynamic time warping is used on the cosine distance measure to find the similarity measure between videos. A 1-nearest neighbor (NN) classifier is used to find the respective crowd behavior attribute classes from the normal videos. Experimental results on large crowd behavior video database show the superior performance of our proposed framework as compared to the baseline and current state-of-the-art methodologies for the crowd behavior recognition task.

The goal of video segmentation is to turn video data into a set of concrete motion clusters that can be easily interpreted as building blocks of the video. There are some works on similar topics like detecting scene cuts in a video, but there is few specific research on clustering video data into the desired number of compact segments. It would be more intuitive, and more efficient, to work with perceptually meaningful entity obtained from a low-level grouping process which we call it `superframe'. This paper presents a new simple and efficient technique to detect superframes of similar content patterns in videos. We calculate the similarity of content-motion to obtain the strength of change between consecutive frames. With the help of existing optical flow technique using deep models, the proposed method is able to perform more accurate motion estimation efficiently. We also propose two criteria for measuring and comparing the performance of different algorithms on various databases. Experimental results on the videos from benchmark databases have demonstrated the effectiveness of the proposed method.

Classification of plants based on a multi-organ approach is very challenging. Although additional data provide more information that might help to disambiguate between species, the variability in shape and appearance in plant organs also raises the degree of complexity of the problem. Despite promising solutions built using deep learning enable representative features to be learned for plant images, the existing approaches focus mainly on generic features for species classification, disregarding the features representing plant organs. In fact, plants are complex living organisms sustained by a number of organ systems. In our approach, we introduce a hybrid generic-organ convolutional neural network (HGO-CNN), which takes into account both organ and generic information, combining them using a new feature fusion scheme for species classification. Next, instead of using a CNN-based method to operate on one image with a single organ, we extend our approach. We propose a new framework for plant structural learning using the recurrent neural network-based method. This novel approach supports classification based on a varying number of plant views, capturing one or more organs of a plant, by optimizing the contextual dependencies between them. We also present the qualitative results of our proposed models based on feature visualization techniques and show that the outcomes of visualizations depict our hypothesis and expectation. Finally, we show that by leveraging and combining the aforementioned techniques, our best network outperforms the state of the art on the PlantClef2015 benchmark. The source code and models are available at https://github.com/cs-chan/Deep-Plant.

Classification of plants based on a multi-organ approach is very challenging. Although additional data provides more information that might help to disambiguate between species, the variability in shape and appearance in plant organs also raises the degree of complexity of the problem. Existing approaches focus mainly on generic features for species classification, disregarding the features representing the organs. In fact, plants are complex entities sustained by a number of organ systems. In our approach, we exploit the PlantClef2015 benchmark, and introduce a hybrid generic-organ convolutional neural network (HGO-CNN), which takes into account both organ and generic information, combining them using a new feature fusion scheme for species classification. We show that our proposed method outperforms the state-of-the-art results.

A fuzzy rough set theory based gait decision system is presented for diagnosis of fall risk. Distracted walking is investigated. Gait cycles of thirty young participants were monitored while they walked across a walkway at a self-selected pace and also while being distracted by performing a task alongside walking. Results shows distracted walking being similar to impaired walking of an elderly person. The aim is to adopt instance selection methods to eliminate redundant or noisy sample from training data thus preventing the system from detecting erroneous gait patterns or deviations. Decision systems have great potential in medical informatics to serve as diagnostic tools. Results show that instance selection improves the efficacy of various classifiers in detecting distracted gait. The model is capable of assessing gait based on the easily obtainable features and categorizing them in terms of fall risk. This will help to identify individuals at risk in fall prevention management.

Presents a listing of the new Associate Editors for this issue of the publication.

This work presents an approach to detect moving objects from Unmanned Aerial Vehicles (UAV). A common framework for most of the existing techniques is using image registration to warp consecutive frames as an ego-motion compensation step and applying frame differencing to detect the moving objects. Assuming a planar scene, we propose the exploitation of telemetry information available from Global Positioning and Inertial Navigation Systems (GPS/INS) to estimate a similarity transformation matrix that would map the image points from one frame to another. In this work, we show that the telemetry-based image registration combined with global registration methods produces more accurate results than the traditional image registration techniques in case of a scene with poor or no texture. To segment the moving objects, we employ the probabilistic background modelling method with mixture of Gaussian distributions.

This paper studies convolutional neural networks (CNN) to learn unsupervised feature representations for 44 different plant species, collected at the Royal Botanic Gardens, Kew, England. To gain intuition on the chosen features from the CNN model (opposed to a ‘black box’ solution), a visualisation technique based on the deconvolutional networks (DN) is utilized. It is found that venations of different order have been chosen to uniquely represent each of the plant species. Experimental results using these CNN features with different classifiers show consistency and superiority compared to the state-of-the art solutions which rely on hand-crafted features.

The papers in this special section focus on the technologies and services that support ambient assisted living (ASL). It is now generally accepted that technology has a role to play in caring for this aging population. The concept of ambient-assisted living (AAL) is a term coined to describe the use of information and communication technologies in people’s daily living and working environment to enable them to stay active longer, remain socially connected, and live independently into old age.

Track let plots (TPs) describe the motion patterns of a small crowd or a large group of people in a given short time span. This feature can be useful in the context of a Bag-of-Words modelling for the recognition of events or actions that unfold in the scene. This work describes a method where evidence from multiple viewpoints is combined. By obtaining this feature for each of the views, and synchronising the available video streams, a feature-level fusion method by concatenation can be effortlessly applied. The presented system is able to recognise specific events in large groups of people from multiple cameras, and to perform equally well as compared to the best single view available. Furthermore, the dimension of the concatenated feature can be reduced by one order of magnitude without loss of performance.

This paper addresses the problem of detecting and localizing abnormal activities in crowded scenes. A spatiotemporal Laplacian eigenmap method is proposed to extract different crowd activities from videos. This is achieved by learning the spatial and temporal variations of local motions in an embedded space. We employ representatives of different activities to construct the model which characterizes the regular behavior of a crowd. This model of regular crowd behavior allows the detection of abnormal crowd activities both in local and global contexts and the localization of regions which show abnormal behavior. Experiments on the recently published data sets show that the proposed method achieves comparable results with the state-of-the-art methods without sacrificing computational simplicity.

In this paper, we describe a model-based behavior analysis system for assisted living. The goal is monitoring the well-being of a single occupant in a home. Behavior is defined as any pattern in a sequence of observations. In analyzing behavior in a smart home, we aim to detect gradual changes in behavior, and atypical (anomalous) behavior. The anomalous behavior may be the result of equipment failure or the result of significant variations in the behavior of the occupant. In the context of a smart home, both situations require human intervention although the response will differ. For the purpose of observing behavior, the smart home is equipped with embedded sensors that unobtrusively record various environmental parameters. Models of behavior are generated from the sensor data. These models are employed to detect trends and infer atypical behavior.

We propose an adaptive tracking system for assisted living that integrates user information about emergency events. Information fusion between user data and visual data is performed in order to estimate and assess the situation at hand. The system is able to dynamically switch between different segmentation and tracking algorithms improving its performance, as shown by the proposed examples.

We investigate the use of artificial local forces governed by physics laws for the spatial formation and coordination of a swarm of VTOL UAVs for ground area surveillance applications. Varying parameter effects are investigated through simulation and a learning algorithm is introduced to optimise the swarms behaviour with respect to self-organising into the optimum formation for a given sized ROI to cover collectively.

The main focus of this special issue is to bring together solutions from research and engineering for the automatic understanding of a complex scene, via a multimodal array of sensors with automation of adaptation as the theme connecting various research areas. Four papers comprise this special issue.

We propose a new method for human action recognition from video streams that is fast and robust to noise and to large changes in camera views. We start by extracting features in the Fourier domain once we obtain the bounding boxes containing the silhouettes of a human for a number of video frames representing a basic action. After preprocessing, we divide each space-time volume into space-time sub-volumes (STSV) and compute their corresponding mean-power spectra as our feature vectors. Our features result in high classification performance even with simple distance measures. We perform an experimental comparison, using the same data, between our method and two state-of-the-art methods.

The objective is to detect activities taking place in a home and to create a model of behavior for the occupant. A behavior is a pattern in the sequence of activities. An array of sensors captures the status of appliances. Models for the occupant's activities are built from the captured data using supervised and unsupervised learning techniques. The models of behavior are built using the hidden Markov model (HMM) technique. Predictive models can be used in a number of ways: to enhance user experience, to maximize resource usage efficiency, for safety and security. This work focuses on supporting independent living and enhancing quality of life of older persons. The ultimate goal is for the system to distinguish between normal and anomalous behavior. In this paper, we present the results of comparing supervised and unsupervised classification techniques applied to the problem of modeling activity for the purpose of modeling behavior in a home.

The ideas put forward by Donald A. Norman (1999) in his monograph entitled The Invisible Computer can be considered the main source of inspiration of a new research area, called ambient intelligence. Ambient intelligence, commonly abbreviated as AmI, is primarily concerned with human–environment interactions. An environment is seen anthropomorphically, as an intelligent agent able to interact with users, creating for them processes to interpret, inform, communicate and dialogue (Abowd & Mynatt, 2000; Remagnino & Foresti, 2005). The history of ambient intelligence starts in Europe in 2001 with the Fifth European Framework Program. At that time, the IST Program Advisory Group (ISTAG) of the European Commission (Directorate General on Information Society and the Media) introduced the concept of ambient intelligence by publishing the report Scenarios for Ambient Intelligence in 2010 (Ducatel et al., 2001). Since then, ambient intelligence has been recognized in Europe as one of the key concepts related to the information and communication technology society. An updated version of the report was published in 2003 under the title Ambient Intelligence: from Vision to Reality (Ducatel et al., 2003). Ambient intelligence's emphasis is on support to human interactions with the environment, user-friendliness, ubiquitous accessibility etc. and requires competences from many research areas, ranging from computer vision, machine learning, distributed computing and middleware, context awareness systems, sensor networks etc. Enticing illustrative scenarios have been published, in which the user wears technology that communicates with systems and devices present in the environment in order to provide information and receive services. Since its inception, ambient intelligence has inspired the design and implementation of system prototypes for intelligent spaces, developed and tested in controlled environments. The scope of this special issue is to publish innovative ideas on a selection of topics related to ambient intelligence. Cameras and computer vision algorithms, for instance, can be used to unobtrusively acquire awareness of the environment. In the paper entitled ‘A multi-camera vision system for fall detection and alarm generation’, Cucchiara et al. propose a system for using cameras to detect people's falls. The use of cameras is preferred in ambient intelligence to accelerometers or other devices because they are not intrusive, people tend to get used to their presence and people's actions are less affected. In this paper, a system of multiple cameras is used to monitor people's movements in house environments and detect changes in people's posture with the specific goal to identify falls. In the paper entitled ‘Understanding intention of movement from electroencephalograms’, Lakany and Conway analyse people's intentions, using electroencephalogram waves. Their method is non-intrusive and it uses a brain–computer interface. Support vector machines are used to select features and perform classification of intentions, and tests are performed to detect the direction of users' movements. While the previous papers address ambient intelligence from the point of view of sensing technologies and algorithms, the following two papers are mainly focused on knowledge representation for context awareness. The paper entitled ‘Knowledge representation for ambient security’ by Snidaro and Foresti proposes an ontology-based methodology for representing the interaction between the user and the environment with specific reference to security scenarios. Similarly, in the paper entitled ‘Context-aware environments: from specification to implementation’ Reigner et al. are concerned with the problem of implementing a context model for a smart environment. Their paper proposes interesting approaches based on ‘networks of situations’, introducing a comparison of the use of Petri nets and hidden Markov models. Finally, in the paper entitled ‘Collection, storage and application of human knowledge in expert system development’ Balch et al. propose an analysis of the knowledge engineering flow, encompassing knowledge acquisition, representation and inference. This flow analysis is presented and applied to the petroleum industry application domain, and specific software tools that use fuzzy logic are utilized.

Abstract: The paper presents the pheromone‐Q‐learning (Phe‐Q) algorithm, a variation of Q‐learning. The technique was developed to allow agents to communicate and jointly learn to solve a problem. Phe‐Q learning combines the standard Q‐learning technique with a synthetic pheromone that acts as a communication medium speeding up the learning process of cooperating agents. The Phe‐Q update equation includes a belief factor that reflects the confidence an agent has in the pheromone (the communication medium) deposited in the environment by other agents. With the Phe‐Q update equation, the speed of convergence towards an optimal solution depends on a number of parameters including the number of agents solving a problem, the amount of pheromone deposit, the diffusion into neighbouring cells and the evaporation rate. The main objective of this paper is to describe and evaluate the performance of the Phe‐Q algorithm. The paper demonstrates the improved performance of cooperating Phe‐Q agents over non‐cooperating agents. The paper also shows how Phe‐Q learning can be improved by optimizing all the parameters that control the use of the synthetic pheromone.

In order to develop a high-level description of events unfolding in a typical surveillance scenario, each successfully tracked event must be classified into type and behaviour. In common with a number of approaches this paper employs a Bayesian classifier to determine type from event attribute such as height, width and velocity. The classifier, however, is extended to integrate all available evidence from the entire track. A not untypical Hidden Markov Model approach has been employed to model the common event behaviours typical of a car-park environment. Both techniques have been probabilistically integrated to generate accurate type and behaviour classifications.

The paper presents an agent based surveillance system for use in monitoring scenes involving both pedestrians and vehicles. The system supplies textual descriptions for the dynamic activity occurring in the 3D world. These are derived by means of dynamic and probabilistic inference based on geometric information provided by a vision system that tracks vehicles and pedestrians. The symbolic scene annotation is given at two major levels of description: the object level and the inter-object level. At object level, each tracked pedestrian or vehicle is assigned a behaviour agent which uses a Bayesian network to infer the fundamental features of the objects' trajectory, and continuously updates its textual description. The inter-object interaction level is interpreted by a situation agent which is created dynamically when two objects are in close proximity. In the work included here the situation agent can describe a two-object interaction in terms of basic textual annotations, to summarise the dynamics of the local action.

International audience