Advancements in remote sensing technology allowed for collecting vast amounts of satellite and aerial imagery with up to 1 cm pixel resolutions, stored in raster format crucial for various research fields. However, processing this data poses challenges, including resolving data dependencies when location, resolution, and coordinate systems do not align and managing large datasets within memory constraints. This paper introduces RDPro, a novel Spark-based system that efficiently processes and analyzes large raster datasets. RDPro features a new data model tailored for data dependencies in a distributed, shared-nothing environment, complete with tools for loading and writing raster data. It also optimizes core raster operations within Spark, allowing users to integrate complex data science workflows. Comparative analysis shows RDPro outperforms existing systems by up to two orders of magnitude.

With the rising amount of publicly available data, data-driven modeling is becoming increasingly popular. Geospatial data is one of the most important facets that can be combined with virtually all data science real-world applications. However, there is a lack of customized geospatial data that can be used in various data science applications from different domains, e.g., hydrology, political science, climatology, and agriculture. This paper introduces a Spark-based extractor that can extract rich geospatial datasets from OpenStreetMap (OSM). OSM hosts crowd-sourced geospatial data that represent a variety of natural and human-made features, e.g., lakes, buildings, and roads. The size of this data is extremely huge and requires complex processing before being ready to use in data science. The proposed extractor runs on Apache SparkSQL which allows it to scale to the Planet.osm file which spans the entire world. In addition to the extractor, we make the data available in various standard formats, e.g., GeoJSON, CSV, KML, and Shapefile. Furthermore, we host these datasets on UCR-Star which allows users to visually explore these datasets and download any subset of the data for any geospatial region.

This paper introduces the open-source Beast system for scalable exploratory data science on big spatio-temporal data. Beast is based on well-established research and has been released to assist the research community with analyzing big spatio-temporal data. Beast provides a set of extensible components that naturally integrate with Spark to build exploratory data science pipelines. Beast can install in less than a minute on an existing Spark cluster and provides a wide array of features including loading vector and raster data represented in standard file formats, synthetic data generation for benchmarking, load-balanced spatial partitioning, data summarization, interactive visualization, and more. Beast builds on several research projects; its goal is to make all this research widely available to researchers in one integrative and coherent system.

Pre-processing spatial data for machine learning applications often includes combining different datasets into a form usable by the machine learning algorithms. Spatial data is generally available in two representations, raster and vector. The best data science and machine learning applications need to combine multiple datasets of both representations which is a data and compute intensive problem. This paper proposes a formal raster-vector join operator, Raptor Join, that can bridge the gap between raster and vector data. It is modeled as a relational join operator in Spark that can be easily combined with other operators, while also offering the advantage of in-situ processing. To implement the Raptor join operator efficiently, we propose a novel Flash index that has a low memory requirement and can process the entire operation with one data scan. We run an extensive experimental evaluation on large scale satellite data with up-to a trillion pixels, and big vector data with up-to hundreds of millions of segments and billions of points, and show that the proposed method can scale to big data with up-to three orders of magnitude performance gain over baselines.

Spatial data is traditionally represented using two data models, raster and vector. Raster data refers to satellite imagery while vector data includes GPS data, Tweets, and regional boundaries. While there are many real-world applications that need to process both raster and vector data concurrently, state-of-the-art systems are limited to processing one of these two representations while converting the other one which limits their scalability. This paper draws the attention of the research community to the research problems that emerge from the concurrent processing of raster and vector data. It describes three real-world applications and explains their computation and access patterns for raster and vector data. Additionally, it runs an extensive experimental evaluation using state-of-the-art big spatial data systems with raster data of up-to a trillion pixels, and vector data with up-to hundreds of millions of edges. The results show that while most systems can analyze raster and vector concurrently, but they have limited scalability for large-scale data.

There has been an increase in the amount of spatial data in the recent years due to the advancements in remote sensing technology and the widespread use of smart phones and GPS technology. This has resulted in petabytes of satellite imagery as well as highly accurate geographical features such as city boundaries, roads, and others being made publicly available. Spatial data can generally be modeled in two representations: raster and vector. Satellite imagery is an example of raster data and is usually represented in form of multi-dimensional arrays. Vector data is represented as a set of points, lines, and polygons, and is used to represent geographical features such as regional boundaries.

Recent advancements in remote sensing technology have resulted in petabytes of data in raster format. This data is often processed in combination with high resolution vector data that represents, for example, city boundaries. One of the common operations that combine big raster and vector data is the zonal statistics which computes some statistics for each polygon in the vector dataset. This paper proposes a novel distributed system to solve the zonal statistics problem which can scale to petabytes of raster and vector data. The proposed method does not require any preprocessing or indexing which makes it perfect for ad-hoc queries that scientists usually want to run. We devise a theoretical cost model that proves the efficiency of our algorithm over the baseline method. Furthermore, we run an extensive experimental evaluation on large scale satellite data with up-to a trillion pixels, and big vector data with up-to hundreds of millions of edges, and we show that our method can perfectly scale to big data with up-to two orders of magnitude performance gain over Rasdaman and Google Earth Engine.

With the increase in amount of remote sensing data, there have been efforts to efficiently process it to help ecologists and geographers answer queries. However, they often need to process this data in combination with vector data, for example, city boundaries. Existing efforts require one dataset to be converted to the other representation, which is extremely inefficient for large datasets. In this demonstration, we focus on the zonal statistics problem, which computes the statistics over a raster layer for each polygon in a vector layer. We demonstrate three approaches, vector-based, raster-based, and raptor-based approaches. The latter is a recent effort of combining raster and vector data without a need of any conversion. This demo will allow users to run their own queries in any of the three methods and observe the differences in their performance depending on different raster and vector dataset sizes.

The recent advances in remote sensing technology resulted in peta bytes of data in raster format. To process this data, it is often combined with high resolution vector data that represents, for example, region boundaries. One of the common operations that combine big vector and raster data is the zonal statistics which computes some aggregate values for each polygon in the vector dataset. This paper proposes a novel and scalable algorithm for zonal statistics that can scale to peta bytes of raster and vector data. The proposed method does not require any preprocessing or indexing making it perfect for ad-hoc queries that scientists usually want to run. We implement a prototype for the proposed method and the initial preliminary results show that the proposed method can scale up-to a trillion pixels.