Time series prediction has wide applications ranging from stock price prediction, product demand estimation to economic forecasting. In this article, we treat the taxi and Uber demand in each location as a time series, and reduce the taxi and Uber demand prediction problem to a time series prediction problem. We answer two key questions in this area. First, time series have different temporal regularity. Some are easy to be predicted and others are not. Given a predictive algorithm such as LSTM (deep learning) or ARIMA (time series), what is the maximum prediction accuracy that it can reach if it captures all the temporal patterns of that time series? Second, given the maximum predictability, which algorithm could approach the upper bound in terms of prediction accuracy? To answer these two question, we use temporal-correlated entropy to measure the time series regularity and obtain the maximum predictability. Testing with 14 million data samples, we find that the deep learning algorithm is not always the best algorithm for prediction. When the time series has a high predictability a simple Markov prediction algorithm (training time 0.5s) could outperform a deep learning algorithm (training time 6 hours). The predictability can help determine which predictor to use in terms of the accuracy and computational costs. We also find that the Uber demand is easier to be predicted compared the taxi demand due to different cruising strategies as the former is demand driven with higher temporal regularity.

Most humans today have mobile phones. According to the GSMA, there are almost 10 billion mobile connections in the world every day. These devices automatically capture behavioral data from human society and store it in databases around the world. However, data capture has several challenges to deal with, especially if it comes from old sources. Obsolete technologies such as 2G and 3G represent two-thirds of the total devices. To the best of our knowledge, all previous work only eliminates obvious problems in the data or use well-curated data. Eliminating traces in a time series can lead to deviations and biases in further analyses, especially when we are studying small areas or groups of peoples in the city. In this work, we present two algorithms to solve the problem of the Neighboring Network Hit (NNH) and calculate the distributions of trips and traveled distances with greater precision in small areas or groups of peoples. The problem of NNH arises when a mobile device connects to cellular sites other than those defined in the network design, which complicates the analysis of space-time mobility. We use cellular device data from three cities in Chile, obtained from the mobile phone operator and duly anonymized. We compare our results with the Government's Origin and Destination Surveys and use a novel method to generate synthetic data to which errors are added in a controlled manner to evaluate the performance of our solution. We conclude that our algorithms improve results compared to naive methods, increasing the accuracy in the count of trips and, mainly, in the distance distributions.

In this paper, we present a community-focused drug abuse monitoring and supporting system, called DrugTracker, that utilizes social media and geospatial data in near real-time. Through the system, users can: (1) Detect drug abuse risk behaviors from social media platforms, e.g., Twitter; (2) Analyze drug abuse risk behaviors by querying consolidated and live datasets with keywords, spatial entities, and time constraints; and (3) Explore the query results and associated data through a web-based user interface in thematic choropleth, heatmap, and statistical charts. To protect the privacy of the Twitter users, whose data is collected, the system automatically hides the re-identification elements in tweets and aggregates the geo-tags into areas such as census tracts. For the demonstration purpose, our DrugTracker system is populated with a database that contains about 10 million tweets from the year 2017, that were annotated as drug abuse risk behavior positive by our deep learning model.

We contribute by developing GeoMatch as a novel, scalable, and efficient big-data pipeline for large-scale map matching on Apache Spark. GeoMatch improves existing spatial big data solutions by utilizing a novel spatial partitioning scheme inspired by Hilbert space-filling curves. Thanks to the partitioning scheme, GeoMatch can effectively balance operations across different processing units and achieve significant performance gains. We demonstrate the effectiveness of GeoMatch through rigorous and extensive benchmarks that consider data sets containing large-scale urban spatial data sets ranging from 166, 253 to 3.78 billion location measurements. Our results show over 17-fold performance improvements compared to previous works while achieving better processing accuracy than current solutions (97.48%).

In the past two decades, new developments in computing, sensing and crowdsourced data have resulted in an explosion in the availability of quantitative information. The possibilities of analyzing this so-called “big data” to inform research and the decision-making process are virtually endless. In general analyses have to be done across multiple data sets in order to bring out the most value of big data. A first important step is to identify temporal correlations between data sets. Given the characteristics of big data in term of volume and velocity, techniques that identify correlations not only need to be scalable, but also need to help users in ordering the correlation across temporal resolutions so that they can focus on important relationships. There is a large body of work in this area, however, most of them either only deal with small data sets, using a fixed temporal resolution, or does not provide a quantifiable measure of a correlation significance. In this paper, we present a method based on mutual information to identify correlations in large data sets. Discovered correlations are suggested to users in an order based on their significance. Our method supports an adaptive streaming technique that minimizes duplicated computation and is implemented on top of Apache Spark for scalability using big data platforms. We also provide a comprehensive evaluation using real-world data sets from NYC Open Data, and compare our findings against a recent study.

In big cities, taxi service is imbalanced. In some areas, passengers wait too long for a taxi, while in others, many taxis roam without passengers. Knowledge of where a taxi will become available can help us solve the taxi demand imbalance problem. In this paper, we employ a holistic approach to predict taxi demand at high spatial resolution. We showcase our techniques using two real-world data sets, yellow cabs and Uber trips in New York City, and perform an evaluation over 9,940 building blocks in Manhattan. Our approach consists of two key steps. First, we use entropy and the temporal correlation of human mobility to measure the demand uncertainty at the building block level. Second, to identify which predictive algorithm can approach the theoretical maximum predictability, we implement and compare three predictors: the Markov predictor (a probability-based predictive algorithm), the Lempel-Ziv-Welch predictor (a sequence-based predictive algorithm), and the Neural Network predictor (a predictive algorithm that uses machine learning). The results show that predictability varies by building block and, on average, the theoretical maximum predictability can be as high as 83%. The performance of the predictors also vary: the Neural Network predictor provides better accuracy for blocks with low predictability, and the Markov predictor provides better accuracy for blocks with high predictability. In blocks with high maximum predictability, the Markov predictor is able to predict the taxi demand with an 89% accuracy, 11% better than the Neural Network predictor, while requiring only 0.03% computation time. These findings indicate that the maximum predictability can be a good metric for selecting prediction algorithms.

Understanding urban human mobility is crucial for epidemic control, urban planning, traffic forecasting systems and, more recently, various mobile and network applications. Nowadays, a variety of urban human mobility data have been gathered and published. Pervasive GPS data can be collected by mobile phones. A mobile operator can track people's movement in cities based on their cellular network location. This urban human mobility data contains rich knowledge about locations and can help in addressing many urban challenges such as traffic congestion or air pollution problems. In this article, we survey recent literature on urban human mobility from a data mining view: from the data collection and cleaning, to the mobility models and the applications. First, we summarize recent public urban human mobility data sets and how to clean and preprocess such data. Second, we describe recent urban human mobility models and predictors, e.g., the deep learning predictor, for predicting urban human mobility. Third, we describe how to evaluate the models and predictors. We conclude by considering how applications can utilize the mobility models and predictive tools for addressing city challenges.

From tweets to urban data sets, there has been an explosion in the volume of textual data that is associated with both temporal and spatial components. Efficiently evaluating queries over these data is challenging. Previous approaches have focused on the spatial aspect. Some used separate indices for space and text, thus incurring the overhead of storing separate indices and joining their results. Others proposed a combined index that either inserts terms into a spatial structure or adds a spatial structure to an inverted index. These benefit queries with highly-selective constraints that match the primary index structure but have limited effectiveness and pruning power otherwise. We propose a new indexing strategy that uniformly handles text, space and time in a single structure, and is thus able to efficiently evaluate queries that combine keywords with spatial and temporal constraints. We present a detailed experimental evaluation using real data sets which shows that not only our index attains substantially lower query processing times, but it can also be constructed in a fraction of the time required by state-of-the-art approaches.

There are increasing volumes of spatio-temporal data from various sources such as sensors, social networks and urban environments. Analysis of such data requires flexible exploration and visualizations, but queries that span multiple geographical regions over multiple time slices are expensive to compute, making it challenging to attain interactive speeds for large data sets. In this paper, we propose a new indexing scheme that makes use of modern GPUs to efficiently support spatio-temporal queries over point data. The index covers multiple dimensions, thus allowing simultaneous filtering of spatial and temporal attributes. It uses a block-based storage structure to speed up OLAP-type queries over historical data, and supports query processing over in-memory and disk-resident data. We present different query execution algorithms that we designed to allow the index to be used in different hardware configurations, including CPU-only, GPU-only, and a combination of CPU and GPU. To demonstrate the effectiveness of our techniques, we implemented them on top of MongoDB and performed an experimental evaluation using two real-world data sets: New York City's (NYC) taxi data - consisting of over 868 million taxi trips spanning a period of five years, and Twitter posts - over 1.1 billion tweets collected over a period of 14 months. Our results show that our GPU-based index obtains interactive, sub-second response times for queries over large data sets and leads to at least two orders of magnitude speedup over spatial indexes implemented in existing open-source and commercial database systems.

As urban population grows, cities face many challenges related to transportation, resource consumption, and the environment. Ride sharing has been proposed as an effective approach to reduce traffic congestion, gasoline consumption, and pollution. Despite great promise, researchers and policy makers lack adequate tools to assess tradeoffs and benefits of various ride-sharing strategies. Existing approaches either make unrealistic modeling assumptions or do not scale to the sizes of existing data sets. In this paper, we propose a real-time, data-driven simulation framework that supports the efficient analysis of taxi ride sharing. By modeling taxis and trips as distinct entities, our framework is able to simulate a rich set of realistic scenarios. At the same time, by providing a comprehensive set of parameters, we are able to study the taxi ride-sharing problem from different angles, considering different stakeholders' interests and constraints. To address the computational complexity of the model, we describe a new optimization algorithm that is linear in the number of trips and makes use of an efficient indexing scheme, which combined with parallelization, makes our approach scalable. We evaluate our framework and algorithm using real data - 360 million trips taken by 13,000 taxis in New York City during 2011 and 2012. The results demonstrate that our framework is effective and can provide insights into strategies for implementing city-wide ride-sharing solutions. We describe the findings of the study as well as a performance analysis of the model.

The manifest importance of cities and the advent of novel data about them are stimulating interest in both basic and applied “urban science” (Bettencourt et al., 2007 [4]; Bettencourt, 2013 [3]). A central task in this emerging field is to document and understand the “pulse of the city” in its diverse manifestations (e.g., in mobility, energy use, communications, economics) both to define the normal state against which anomalies can be judged and to understand how macroscopic city observables emerge from the aggregate behavior of many individuals (Louail, 2013 [9]; Ferreira et al., 2013 [6]). Here we quantify the dynamics of an urban lightscape through the novel modality of persistent synoptic observations from an urban vantage point. Established astronomical techniques are applied to visible light images captured at 0.1 Hz to extract and analyze the light curves of 4147 sources in an urban scene over a period of 3 weeks. We find that both residential and commercial sources in our scene exhibit recurring aggregate patterns, while the individual sources decorrelate by an average of one hour after only one night. These highly granular, stand-off observations of aggregate human behavior – which do not require surveys, in situ monitors, or other intrusive methodologies – have a direct relationship to average and dynamic energy usage, lighting technology, and the impacts of light pollution. They may also be used indirectly to address questions in urban operations as well as behavioral and health science. Our methodology can be extended to other remote sensing modalities and, when combined with correlative data, can yield new insights into cities and their inhabitants.

Graph analytics on social networks, Web data, and communication networks has been widely used in a plethora of applications. Many graph analytics algorithms are based on breadth-first search (BFS) graph traversal, which is not only time-consuming for large datasets but also involves much redundant computation when executed multiple times from different start vertices. In this paper, we propose Multi-Source BFS (MS-BFS), an algorithm that is designed to run multiple concurrent BFSs over the same graph on a single CPU core while scaling up as the number of cores increases. MS-BFS leverages the properties of small-world networks , which apply to many real-world graphs, and enables efficient graph traversal that: (i) shares common computation across concurrent BFSs; (ii) greatly reduces the number of random memory accesses; and (iii) does not incur synchronization costs. We demonstrate how a real graph analytics application---all-vertices closeness centrality---can be efficiently solved with MS-BFS. Furthermore, we present an extensive experimental evaluation with both synthetic and real datasets, including Twitter and Wikipedia, showing that MS-BFS provides almost linear scalability with respect to the number of cores and excellent scalability for increasing graph sizes, outperforming state-of-the-art BFS algorithms by more than one order of magnitude when running a large number of BFSs.

Data obtained from New York City's Taxi & Limousine Commission highlights the importance of yellow cabs in the city's transportation infrastructure. In 2011, for example, there were nearly 180 million cab rides. In this paper, we provide detailed visualizations of these numerous journeys. Using a random sample, we also develop several relevant economic metrics. For example, we find that nearly half of all rides were taken by a single passenger traveling short distances. In contrast, out-of-pocket costs faced by passengers averaged less than $6 per mile, with 90% paying less than $8 per mile, even when the social value of the city's roads is highest. From a city planner's perspective, the current fare structure for yellow cabs may not be optimal. Finally, we propose a statistical method to analyze dynamic flow patterns of yellow-cab rides that can ultimately be extended to a multivariate model.

We show how work flow systems can be augmented to leverage provenance information to enhance usability. In particular, we will demonstrate new mechanisms and intuitive user interfaces designed to allow users to query work flows by example and to refine work flows by analogies. These techniques are implemented in VisTrails, an open-source provenance-enabled scientific work flow system that can be combined with a wide range of tools, libraries, and visualization systems. We will show di erent scenarios where these techniques can be used to simplify the notoriously hard tasks of creating and refining work flows.

Scientists are now faced with an incredible volume of data to analyze. To successfully analyze and validate various hypothesis, it is necessary to pose several queries, correlate disparate data, and create insightful visualizations of both the simulated processes and observed phenomena. Often, insight comes from comparing the results of multiple visualizations. Unfortunately, today this process is far from interactive and contains many error-prone and time-consuming tasks. As a result, the generation and maintenance of visualizations is a major bottleneck in the scientific process, hindering both the ability to mine scientific data and the actual use of the data. The VisTrails system represents our initial attempt to improve the scientific discovery process and reduce the time to insight. In VisTrails, we address the problem of visualization from a data management perspective: VisTrails manages the data and metadata of a visualization product. In this demonstration, we show the power and flexibility of our system by presenting actual scenarios in which scientific visualization is used and showing how our system improves usability, enables reproducibility, and greatly reduces the time required to create scientific visualizations.