We consider long-term path forecasting problems in crowds, where future sequence trajectories are generated given a short observation. Recent methods for this problem have focused on modeling social interactions and predicting multi-modal futures. However, it is not easy for machines to successfully consider social interactions, such as avoiding collisions while considering the uncertainty of futures under a highly interactive and dynamic scenario. In this paper, we propose a model that incorporates multiple interacting motion sequences jointly and predicts multi-modal socially acceptable distributions of futures. Specifically, we introduce a new aggregation mechanism for social interactions, which selectively models long-term inter-related dynamics between movements in a shared environment through a message passing mechanism. Moreover, we propose a loss function that not only accesses how accurate the estimated distributions of the futures are but also considers collision avoidance. We further utilize mixture density functions to describe the trajectories and learn the multi-modality of future paths. Extensive experiments over several trajectory prediction benchmarks demonstrate that our method is able to forecast socially acceptable distributions in complex scenarios.

Accurate human path forecasting in complex and crowded scenarios is critical for collision avoidance of autonomous driving and social robots navigation. It still remains as a challenging problem because of dynamic human interaction and intrinsic multimodality of human motion. Given the observation, there is a rich set of plausible ways for an agent to walk through the circumstance. To address those issues, we propose a spatio-temporal model that can aggregate the information from socially interacting agents and capture the multimodality of the motion patterns. We use mixture density functions to describe the human path and predict the distribution of future paths with explicit density. To integrate more factors to model interacting people, we further introduce a coordinate transformation to represent the relative motion between people. Extensive experiments over several trajectory prediction benchmarks demonstrate that our method is able to forecast various plausible futures in complex scenarios and achieves state-of-the-art performance.

Recently, owing to the outstanding capability of deep learning in solving ill-posed problems, the single-frame super-resolution (SR) researches tend to focus on deep learning methods largely. However, related researches are implemented and evaluated through various datasets and different deep learning frameworks, which hinders the comparison of performance among different methods and heavily hampers the progress of SR techniques. In this study, we present GeoSR, an open source computer vision package for deep learning based single-frame remote sensing imagery super-resolution to facilitate the development of the SR community. As a unified, simple, and flexible package, GeoSR contains pipeline-like integrated tools from data retrieval to final result evaluation, which enables users to develop self-defined models conveniently; several state-of-the-art models trained through the same high-quality dataset are provided as the baseline in the package as well. Moreover, the proposed package could potentially serve as a viable backend for other related packages such as image segmentation with high efficiency.

Recently, deep learning algorithms, especially fully convolutional network based methods, have become very popular in the field of remote sensing. However, these methods are implemented and evaluated through various datasets and deep learning frameworks. There has not been a package that covers these methods in a unified manner. In this study, we introduce a computer vision package termed Geoseg that focuses on building segmentation and outline extraction. Geoseg implements nine state-of-the-art models as well as utility scripts needed to conduct model training, logging, evaluation, and visualization. The implementation of Geoseg emphasizes unification, simplicity, and flexibility. The performance and computational efficiency of all implemented methods are evaluated by a comparison experiment using a unified, high-quality aerial image dataset.

The automatic digitizing of paper maps is a significant and challenging task for both academia and industry. As an important procedure of map digitizing, the semantic segmentation section is mainly relied on manual visual interpretation with low efficiency. In this study, we select urban planning maps as a representative sample and investigate the feasibility of utilizing U-shape fully convolutional based architecture to perform end-to-end map semantic segmentation. The experimental results obtained from the test area in Shibuya district, Tokyo, demonstrate that our proposed method could achieve a very high Jaccard similarity coefficient of 93.63% and an overall accuracy of 99.36%. For implementation on GPGPU and cuDNN, the required processing time for the whole Shibuya district can be less than three minutes. The results indicate the proposed method can serve as a viable tool for urban planning map semantic segmentation task with high accuracy and efficiency.

We categorize this research in terms of its contribution to both graph theory and computer vision. From the theoretical perspective, this study can be considered as the first attempt to formulate the idea of mining maximal frequent subgraphs in the challenging domain of messy visual data, and as a conceptual extension to the unsupervised learning of graph matching. We define a soft attributed pattern (SAP) to represent the common subgraph pattern among a set of attributed relational graphs (ARGs), considering both their structure and attributes. Regarding the differences between ARGs with fuzzy attributes and conventional labeled graphs, we propose a new mining strategy that directly extracts the SAP with the maximal graph size without applying node enumeration. Given an initial graph template and a number of ARGs, we develop an unsupervised method to modify the graph template into the maximal-size SAP. From a practical perspective, this research develops a general platform for learning the category model (i.e., the SAP) from cluttered visual data (i.e., the ARGs) without labeling "what is where," thereby opening the possibility for a series of applications in the era of big visual data. Experiments demonstrate the superior performance of the proposed method on RGB/RGB-D images and videos.

The understanding of mass population movements has greatly advanced with the rapid spread of ubiquitous devices. Anonymized call detail records (CDRs) for mobile phones have enabled us to not only trace individual trajectories but also approximate activity patterns, including significant locations such as homes and workplaces. The majority of studies analyzing CDRs attempt to utilize the mobility patterns of anonymized crowds to improve transportation and public health. This is quite reasonable because CDRs can capture the movements of people at given times and places, whereas general statistics usually account for a population based on their locations of residence. However, it has also been pointed out that there are discrepancies between the movements of people as observed through CDRs and those of an entire population in a given area. This is because CDRs only represent device users. In fact, we can never learn about the population that is unobservable through CDRs only by analyzing CDRs. Therefore, this study attempts to provide clues to help us understand the whereabouts of the unobservable population by analyzing two months of the CDRs for 58 volunteers with mobile device service from a major telecommunications company in combination with field survey data from Dhaka. We surveyed the personal and household attributes of mobile users in relation to their calling behavior. The analysis results show that per mobile user observed in CDRs, there is an average of roughly 2.4 to 2.8 unobservable people. Their age groups and gender composition are also provided. We find that male and female users exhibit opposite trends in call locations according to the presence of children within the household. In addition, based on field observations, we find that the location and time distributions of small children follow some specific routines. Our findings contribute to the understanding of the whereabouts of the unobservable population, the majority of whom are children and are considered to be vulnerable to disasters or infectious diseases but are difficult to locate through CDRs alone.

Mining object-level knowledge, that is, building a comprehensive category model base, from a large set of cluttered scenes presents a considerable challenge to the field of artificial intelligence. How to initiate model learning with the least human supervision (i.e., manual labeling) and how to encode the structural knowledge are two elements of this challenge, as they largely determine the scalability and applicability of any solution. In this article, we propose a model-learning method that starts from a single-labeled object for each category, and mines further model knowledge from a number of informally captured, cluttered scenes. However, in these scenes, target objects are relatively small and have large variations in texture, scale, and rotation. Thus, to reduce the model bias normally associated with less supervised learning methods, we use the robust 3D shape in RGB-D images to guide our model learning, then apply the properly trained category models to both object detection and recognition in more conventional RGB images. In addition to model training for their own categories, the knowledge extracted from the RGB-D images can also be transferred to guide model learning for a new category, in which only RGB images without depth information in the new category are provided for training. Preliminary testing shows that the proposed method performs as well as fully supervised learning methods.

3D reconstruction from a single image is a classical problem in computer vision. However, it still poses great challenges for the reconstruction of daily-use objects with irregular shapes. In this paper, we propose to learn 3D reconstruction knowledge from informally captured RGB-D images, which will probably be ubiquitously used in daily life. The learning of 3D reconstruction is defined as a category modeling problem, in which a model for each category is trained to encode category-specific knowledge for 3D reconstruction. The category model estimates the pixel-level 3D structure of an object from its 2D appearance, by taking into account considerable variations in rotation, 3D structure, and texture. Learning 3D reconstruction from ubiquitous RGB-D images creates a new set of challenges. Experimental results have demonstrated the effectiveness of the proposed approach.

Graph matching and graph mining are two typical areas in artificial intelligence. In this paper, we define the soft attributed pattern (SAP) to describe the common subgraph pattern among a set of attributed relational graphs (ARGs), considering both the graphical structure and graph attributes. We propose a direct solution to extract the SAP with the maximal graph size without node enumeration. Given an initial graph template and a number of ARGs, we modify the graph template into the maximal SAP among the ARGs in an unsupervised fashion. The maximal SAP extraction is equivalent to learning a graphical model (i.e. an object model) from large ARGs (i.e. cluttered RGB/RGB-D images) for graph matching, which extends the concept of "unsupervised learning for graph matching." Furthermore, this study can be also regarded as the first known approach to formulating "maximal graph mining" in the graph domain of ARGs. Our method exhibits superior performance on RGB and RGB-D images.

A large category model base can provide object-level knowledge for various perception tasks of the intelligent vehicle system. The automatic and efficient construction of such a model base is highly desirable but challenging. This paper presents a novel semi-supervised approach to discover possible prototype models of 3D object structures from the point cloud of a large and complex environment, given a limited number of seeds in an object category. Our method incrementally trains the models while simultaneously collecting object samples. Considering the bias problem of model learning caused by bias accumulation in a sample collection, we propose to gradually differentiate the standard category model into several sub-category models to represent different intra-category structural styles. Thus, new sub-categories are discovered and modeled, old models are improved, and redundant models for similar structures are deleted iteratively during the learning process. This multiple-model strategy provides several interactive options for the category boundary to deal with the bias problem. Experimental results demonstrate the effectiveness and high efficiency of our approach to model mining from “big point cloud data”.

Mobile phones are arguably one of the most prolific sources of large-scale human mobility data. The availability of this data has generated a massive body of research focused on understanding the dynamics and patterns of human mobility. However, it is increasingly evident that additional value can be derived from such data. This paper proposes a novel approach for understanding the attributes of mobile users by analyzing calling behavior derived from field survey data, in combination with call detail records (CDRs). Our survey reveals distinctive traits in calling behavior that correspond to user attributes. Analysis results demonstrate that frequent call locations, the variability in call time distributions, and the locations from which calls are made around midday are all keys to distinguishing gender. In addition, the location of calls initiated during the morning hours is a key to analyzing income levels for males.

Registering multiple views is an essential and challenging problem for many intelligent transportation applications that employ a mobile sensing platform or consist of multiple stationary sensors. In this paper a novel algorithm is presented for multiple-view registration under a loop closure constraint. Different from most existing methods, which use general optimization techniques, our method studies the mechanism of adjusting the poses of views in a loop and provides a highly efficient and accurate solution. We prove that translation vectors can be decoupled if the same point set is used in each view to associate the previous and subsequent views, leading to our solution for such decouplable cases. If this condition does not hold, an exact solution of translation vectors is provided when rotation parameters are given, which results in our iterative solution for general cases by updating rotation and translation alternately. In our method, the effect of the accumulated pose error in a loop can be distributed to all views efficiently through loop factors, and only a few iterations are needed. Most important of all, in each iteration our method has linear computational complexity with respect to the number of views, which is much superior to that of state-of-the-art methods. A series of experiments was conducted, involving simulation of thousands of views and real vehicle-borne sensing data that include 65 371 point pairs in 352 views. Experimental results show that our proposed method is not only stable and highly efficient but also provides competitive accuracy relative to existing methods.

An object model base that covers a large number of object categories is of great value for many computer vision tasks. As artifacts are usually designed to have various textures, their structure is the primary distinguishing feature between different categories. Thus, how to encode this structural information and how to start the model learning with a minimum of human labeling become two key challenges for the construction of the model base. We design a graphical model that uses object edges to represent object structures, and this paper aims to incrementally learn this category model from one labeled object and a number of casually captured scenes. However, the incremental model learning may be biased due to the limited human labeling. Therefore, we propose a new strategy that uses the depth information in RGBD images to guide the model learning for object detection in ordinary RGB images. In experiments, the proposed method achieves superior performance as good as the supervised methods that require the labeling of all target objects.

Although graph matching is a fundamental problem in pattern recognition, and has drawn broad interest from many fields, the problem of learning graph matching has not received much attention. In this paper, we redefine the learning of graph matching as a model learning problem. In addition to conventional training of matching parameters, our approach modifies the graph structure and attributes to generate a graphical model. In this way, the model learning is oriented toward both matching and recognition performance, and can proceed in an unsupervised fashion. Experiments demonstrate that our approach outperforms conventional methods for learning graph matching.

The building of an object-level knowledge base is the foundation of a new methodology for many perception tasks in artificial intelligence, and is an area that has received increasing attention in recent years. In this paper, we propose, for the first time, to mine category shape patterns directly from a large urban environment, thus constructing a category structure base. Conventionally, category patterns are learned from a large collection of object samples, but automatic object collection requires prior knowledge of category structures. To solve this chicken-and-egg problem, we learn shape patterns from raw segmentations, and then refine these segmentations based on the pattern knowledge. In the process, we solve two challenging problems of knowledge mining. First, as some categories have large intra-category structure variations, we design an entropy-based method to determine the structure variation for each category, in order to establish the correct range of sample collection. Second, because incorrect segmentation is unavoidable without prior knowledge, we propose a novel unsupervised method that uses a pattern competition strategy to identify and subtract shape patterns formed by incorrectly segmented objects. This ensures that shape patterns are meaningful at the object level. Experimental results demonstrated the effectiveness of the proposed method for category structure mining in a large urban environment.

For reasons of public security, an intelligent surveillance system that can cover a large, crowded public area has become an urgent need. In this article, we propose a novel laser-based system that can simultaneously perform tracking, semantic scene learning, and abnormality detection in a fully online and unsupervised way. Furthermore, these three tasks cooperate with each other in one framework to improve their respective performances. The proposed system has the following key advantages over previous ones: (1) It can cover quite a large area (more than 60×35m), and simultaneously perform robust tracking, semantic scene learning, and abnormality detection in a high-density situation. (2) The overall system can vary with time, incrementally learn the structure of the scene, and perform fully online abnormal activity detection and tracking. This feature makes our system suitable for real-time applications. (3) The surveillance tasks are carried out in a fully unsupervised manner, so that there is no need for manual labeling and the construction of huge training datasets. We successfully apply the proposed system to the JR subway station in Tokyo, and demonstrate that it can cover an area of 60×35m, robustly track more than 150 targets at the same time, and simultaneously perform online semantic scene learning and abnormality detection with no human intervention.

Multitarget tracking becomes significantly more challenging when the targets are in close proximity or frequently interact with each other. This article presents a promising online system to deal with these problems. The novelty of this system is that laser and vision are integrated with tracking and online learning to complement each other in one framework: when the targets do not interact with each other, the laser-based independent trackers are employed and the visual information is extracted simultaneously to train some classifiers online for “possible interacting targets”. When the targets are in close proximity, the classifiers learned online are used alongside visual information to assist in tracking. Therefore, this mode of cooperation not only deals with various tough problems encountered in tracking, but also ensures that the entire process can be completely online and automatic. Experimental results demonstrate that laser and vision fully display their respective advantages in our system, and it is easy for us to obtain a good trade-off between tracking accuracy and the time-cost factor.

Abnormal activity detection plays a crucial role in surveillance applications, and a surveillance system that can perform robustly in the extremely crowded area has become an urgent need for public security. In this paper, we propose a novel laser-based system which can simultaneously perform the tracking, semantic scene learning and abnormality detection in the large and crowded environment. In our system, a novel abnormality detection model is proposed, and it considers and combines various factors that will influence human activity. Moreover, this model intensively investigate the relationship between pedestrians' social behaviors and their walking scenarios. We successfully applied the proposed system to the JR subway station of Tokyo, which can cover a 60×35m area, robustly track more than 180 targets at the same time and simultaneously perform the online semantic scene learning and abnormality detection with no human intervention.

Abnormal activity detection plays a crucial role in surveillance applications, and such system has become an urgent need for public security. In this paper, we propose a novel laser-based system, which can perform the online detection of abnormal activity with an unsupervised way. The proposed system has the following key features that make it advantageous over previous ones: (1) It can cover quite a large and crowded area, such as subway station, public square, intersection and etc. (2) The overall system can vary with time period, incrementally learn the behavior pattern of pedestrians and perform the fully online detection of abnormal activity. This feature makes our system be quite suitable for the real-time applications. (3) The abnormal activity detection is carried out with a fully unsupervised way, there is no need for manual labelling and constructing the huge training datasets. We successfully applied the proposed system into the JR subway station of Tokyo, which can cover a 60×35m area, track more 150 targets at the same time and simultaneously perform the robust detection of abnormal activity with no human intervention.

In this study, we present a novel system for crowd surveillance and quantified analysis based on laser range scanners. By mounting a laser scanner at a swinging platform, the spatial information of passengers inside the area of interest can be reconstructed in a form of 3D points. Multiple laser scanners are integrated together by semi-auto calibration procedures. Background map is generated through histogram analysis of scan maps, and is further applied for 3D moving object detection. An improved version of mean-shift clustering algorithm is proposed to extract individual passengers efficiently. In addition, quantified crowdness analysis is conducted from different aspects to indicate the situation inside the surveillance area according to the extraction results of passengers. The proposed system was tested in a central subway station in Tokyo and experimental results demonstrate the effectiveness of our proposed system.

Learning the knowledge of scene structure and tracking a large number of targets are both active topics of computer vision in recent years, which plays a crucial role in surveillance, activity analysis, object classification and etc. In this paper, we propose a novel system which simultaneously performs the Learning-Semantic-Scene and Tracking, and makes them supplement each other in one framework. The trajectories obtained by the tracking are utilized to continually learn and update the scene knowledge via an online un-supervised learning. On the other hand, the learned knowledge of scene in turn is utilized to supervise and improve the tracking results. Therefore, this “adaptive learning-tracking loop” can not only perform the robust tracking in high density crowd scene, dynamically update the knowledge of scene structure and output semantic words, but also ensures that the entire process is completely automatic and online. We successfully applied the proposed system into the JR subway station of Tokyo, which can dynamically obtain the semantic scene structure and robustly track more than 150 targets at the same time.

Multi-target tracking becomes significantly more challenging when the targets are in close proximity or frequently interact with each other. This paper presents a promising tracking system to deal with these problems. The novelty of this system is that laser and vision, tracking and learning are integrated and can complement each other in one framework: when the targets do not interact with each other, the laser-based independent trackers are employed and the visual information is extracted simultaneously to train some classifiers for the “possible interacting targets”. When the targets are in close proximity, the learned classifiers and visual information are used to assist in tracking. Therefore, this mode of co-operation between them not only deals with various tough problems encountered in the tracking, but also ensures that the entire process can be completely on-line and automatic. Experimental results demonstrated that laser and vision fully display their respective advantages in our system, and it is easy for us to obtain a perfect trade-off between tracking accuracy and time-cost.

We propose a novel system for extracting multiple people in crowded scenes by employing LED 3D range sensor. This new kind of device can capture depth image at a relatively high frame speed, which makes the extraction of dynamic objects possible. However, it suffers from the strong noise caused by the environment illumination. Here we propose a novel method for multiple people extraction by integrating an improved version of mean-shift clustering algorithm and total variation based denoising technique. When handling the depth image, it is considered both as a point cloud as well as a 2D image. In this way, different properties of the depth image are sufficiently exploited. The proposed method can fully automatically extract multiple people in a cluttered environment, even from a mobile platform. The experiment conducted at the platform of a railway station demonstrates the effectiveness of our proposed algorithm.

This work is motivated by the following two potential applications: 1) enhancing driving safety and 2) collecting traffic data in a large dynamic urban environment. A laser-scanner-based approach is proposed. The problem is formulated as a simultaneous localization and mapping (SLAM) with object tracking and classification, where the focus is on managing a mixture of data from both dynamic and static objects in a highly dynamic environment. A trajectory-oriented closure is also proposed using the sporadically available global positioning system (GPS) measurements in urban areas to assist for global accuracy, particularly when the vehicle makes a noncyclical measurement in a large outdoor environment. Experiments are conducted using the data that were collected along a course near 4.5 km in a highly dynamic environment. Possibilities of the approaches toward the two potential applications are demonstrated, and avenues for future works are discussed.

In this research, we propose a method of SLAM in a dynamic large outdoor environment using a laser scanner. Focus are cast on solving two major problems: 1) achieving global accuracy especially in non-cyclical environment, 2) tackling a mixture of data from both dynamic and static objects. Algorithms are developed, where GPS data and control inputs are used to diagnose pose error and guide to achieve a global accuracy; Classification of laser points and objects are conducted not in an independent module but across the processing in a framework of SLAM with moving object detection and tracking. Experiments are conducted using the data from two test-bed vehicles, and performance of the algorithms are demonstrated.

We propose a novel system for tracking a variable number of pedestrians in crowded scenes by exploiting laser range scanners. Based on the specific pattern generated by walking feet in the spatio-temporal domain, a walking model is constructed and applied to track pedestrians. To track interactive targets, an algorithm based on Interactive Multiple Particle Filters (IMPF) is proposed, whose computation load increases linearly with the number of targets. To handle a variable number of pedestrians, spatio-temporal correlation analysis in combination with a mean shift based clustering technique is proposed. Compared with camera-based surveillance methods, our system provides a novel technique for automatically tracking a large number of pedestrians in a relatively large area. The experiments, in which over 2600 pedestrians were tracked in 10 minutes at a 60 m × 20 m subway station, show the effectiveness of our proposed algorithm.

In this research, we proposed a system of detecting and monitoring pedestrians' motion trajectories at a populated and wide environment, such as exhibition hall, supermarket etc., using the horizontally profiling single-row laser range scanners on a mobile platform. A simplified walking model is defined to track the rhythmic swing feet at the ground level. Pedestrians are recognized by detecting the braided styles, which is a typical appearance that could discriminate the data of moving feet with other mobile and motionless objects. Two experiments are conducted. One is at the laboratory environment, the purpose of which is to examine the algorithm in details. Another is at an exhibition hall, a populated and wide environment, the purpose is to examine whether the system could be applied for practical needs. It is a big challenge, while the system did well. Pedestrians in the exhibition hall at the moment of measurement are detected. Their motion trajectories are extracted, and associated to the background map, which is made of the motionless objects, and covers the whole exhibition hall.

We propose a novel system for tracking multiple pedestrians in a crowded scene by exploiting single-row laser range scanners that measure distances of surrounding objects. A walking model is built to describe the periodicity of the movement of the feet in the spatial-temporal domain, and a mean-shift clustering technique in combination with spatial- temporal correlation analysis is applied to detect pedestrians. Based on the walking model, particle filter is employed to track multiple pedestrians. Compared with camera-based methods, our system provides a novel technique to track multiple pedestrians in a relatively large area. The experiments, in which over 300 pedestrians were tracked in 5 minutes, show the validity of the proposed system.

We propose a novel system for analyzing pedestrians' walking patterns by exploiting single-row laser range scanners that measure distances of surrounding objects by reflecting eye-safe laser beams. A walking model is built in the spatial-temporal domain to describe the periodicity of the movement of the feet, and a mean-shift technique is applied to recover model parameters. Compared with camera-based methods, our system provides a novel technique to analyze the behavior of pedestrians. The experiments show the validity of the algorithm.