Data scientists and researchers utilize enormous spatio-temporal data and build machine learning models to solve practical problems in diverse domains including intelligent transportation, urban planning, epidemic prediction, and many more. Extracting application-specific features from big spatio-temporal data poses system requirements of heterogeneous data support, efficient and scalable computing over spatial and temporal dimensions, as well as a user-friendly programming interface. This paper presents ST4ML, a distributed spatio-temporal data processing system to support scalable machine-learning-oriented applications. We propose a three-stage pipelining computing framework, namely "selection-conversion-extraction" to abstract the distributed computing flow and implement it based on Apache Spark. To the best of our knowledge, ST4ML is the first of its kind to realize our design considerations. Extensive experiments with real-world datasets evidence that ST4ML outperforms straightforward extensions of existing ST data processing systems by up to an order of magnitude. ST4ML is open-sourced at https://github.com/Panrong/st4ml.

While urban rail transit systems are playing an increasingly important role in meeting the transportation demands of people, precise awareness of how the human crowd is distributed within such a system is highly necessary, which serves a range of important applications including emergency response, transit recommendation, and commercial valuation. Most rail transit systems are closed systems where once entered the passengers are free to move around all stations and are difficult to track. In this article, we attempt to estimate the crowd distribution based only on the tap-in and tap-out records of all the rail riders. Specifically, we study Singapore MRT (Mass Rapid Transit) as a vehicle and leverage EZ-Link transit card records to estimate the crowd distribution. Guided by a key observation that the passenger inflows and arrival flows at different MRT stations and time are spatio-temporally correlated due to behavioral consistency of MRT riders, we design and implement a machine learning-based solution, CrowdAtlas, that captures MRT riders’ transition probabilities among stations and across time, and based on that accurately estimates the crowd distribution within the MRT system. Our comprehensive performance evaluations with both trace-driven studies and real-world experiments in MRT disruption cases demonstrate the effectiveness of CrowdAtlas.

While the urban rail transit systems are playing an increasingly important role in meeting the transportation demands of people, the precise awareness of how the human crowd is distributed within the urban rail transit system is highly necessary, which serves to a range of important applications including emergency response, transit recommendation, commercial valuation, etc. Most urban rail transit systems are closed systems where once entered the travelers are free to move around all stations that are connected into the system and are difficult to track. In this paper, we attempt to estimate the crowd distribution within the urban rail transit system based only on the entrance and exit records of all the rail riders. Specifically, we study Singapore MRT (Mass Rapid Transit) as a vehicle and leverage the tap-in and tap-out records of the EZ-Link transit cards to estimate the crowd distribution. Guided by a key observation that the passenger inflows and arrival flows at various MRT stations are spatio-temporally correlated due to behavioral consistence of MRT riders, we design and implement a machine learning based solution, CrowdAtlas, that accurately estimates the crowd distribution within the MRT system. Our trace-driven performance evaluation demonstrates the effectiveness of CrowdAtlas.

Service disruptions of rail transit systems become more frequent in the past decades in urban cities like Singapore, due to various reasons such as power failures, signal errors, etc. We study and predict the impact of disruptions to transit systems and commuters. This benefits service providers in making both short and long term plans to improve their services. Specifically, we define two metrics, stay ratio and travel delay, to quantify the impact. To tackle the main challenge of abnormal data scarcity, i.e., only 6 observed disruptions in our one-year data records, we propose to format the problem into a training problem on a feature space relevant to alternative route choices of the commuters. We demonstrate the new feature space corresponds to more similar data distribution among different disruptions, which is beneficial for training more generalisable predictors for future disruptions. We implement and evaluate our approach with a real-world transit card dataset. The result clearly shows that our method outperforms a range of baseline methods.

This paper presents TrajGen, an approach to generate artificial datasets of mobility trajectories based on an original trajectory dataset while retaining the utility of the original data in supporting various mobility applications. The generated mobility data is disentangled with the original data and can be shared without compromising the data privacy. TrajGen leverages Generative Adversarial Nets combined with a Seq2Seq model to generate the spatial-temporal trajectory data. TrajGen is implemented and evaluated with real-world taxi trajectory data in Singapore. The extensive experimental results demonstrate that TrajGen is able to generate artificial trajectory data that retain key statistical characteristics of the original data. Two case studies, i.e. road map updating and Origin-Destination demand estimation are performed with the generated artificial data, and the results show that the artificial trajectories generated by TrajGen retain the utility of original data in supporting the two applications.

In this paper, we study the set reconciliation problem, in which each member of a node pair has a set of objects and seeks to deliver its unique objects to the other member. How could each node compute the set difference, however, is challenging in the set reconciliation problem. To address such an issue, we propose a lightweight but efficient method that only requires the pair of nodes to represent objects using a counting Bloom filter (CBF) of size $(O(d))$ and exchange with each other, where $(d)$ denotes the total size of the set differences. A receiving node then subtracts the received CBF from its local one via minus operation proposed in this paper. The resultant CBF can approximately represent the union of the set differences and thus the set difference to each node can be identified after querying the resultant CBF. In this paper, we propose a novel estimator through which each node can accurately estimate not only the value of $(d)$ but also the size of the set difference to each node. Such an estimation result can be used to optimize the parameter setting of the CBF to achieve less false positives and false negatives. Comprehensive analysis and evaluation demonstrates that our method is more efficient than prior BF-based methods in terms of achieving the same accuracy with less communication cost. Moreover, our reconciliating method needs no prior context logs and it is very useful in networking and distributed applications.

Contour mapping is a crucial part of many wireless sensor network applications. Many efforts have been made to avoid collecting data from all the sensors in the network and producing maps at the sink, which is proven to be inefficient. The existing approaches (often aggregation based), however, suffer from heavy transmission traffic and incur large computational overheads on each sensor node. We propose Iso-Map, an energy-efficient protocol for contour mapping, which builds contour maps based solely on the reports collected from intelligently selected “isoline nodes” in wireless sensor networks. Iso-Map achieves high-quality contour mapping while significantly reducing the generated traffic from O(n) to O(\sqrt n), where n is the total number of sensor nodes in the field. The pernode computation overhead is also restrained as a constant. We conduct comprehensive trace-driven simulations to verify this protocol, and demonstrate that Iso-Map outperforms the previous approaches in the sense that it produces contour maps of high fidelity with significantly reduced energy cost.

Event detection is a crucial task for wireless sensor network applications, especially environment monitoring. Existing approaches for event detection are mainly based on some predefined threshold values, and thus are often inaccurate and incapable of capturing complex events. For example, in coal mine monitoring scenarios, gas leakage or water osmosis can hardly be described by the overrun of specified attribute thresholds, but some complex pattern in the full-scale view of the environmental data. To address this issue, we propose a non-threshold based approach for the real 3D sensor monitoring environment. We employ energy-efficient methods to collect a time series of data maps from the sensor network and detect complex events through matching the gathered data to spatio-temporal data patterns. Finally, we conduct trace driven simulations to prove the efficacy and efficiency of this approach on detecting events of complex phenomena from real-life records.