Remote sensing data has become increasingly important for domain scientists from diverse areas including agriculture. Traditional data analytics methods cannot scale to big-data as they normally run on a single machine. This led to new systems that can process satellite data on distributed environments. However, directly translating traditional query processing pipelines to distributed systems is still challenging due to the complexity of raster data processing, such as varying spatial and temporal resolutions, computational overhead from iterative operations, and dealing with excessive data volumes. This paper introduces FieldSAT, a real-life case study of scaling query pipelines from the agriculture field using a Spark-based raster processing system named RDPro. This work focuses on the BAITSSS model, which is a complex, yet very important, model for estimating evapotranspiration from raster data on large scale. The paper goes beyond straightforward implementation of the model and introduces key optimization techniques and lessons learned that can be broadly applied to similar applications. The optimizations introduced in FieldSAT include: managing prolonged Spark DAGs and dependency chains, minimizing shuffle volumes, implementing raster interpolation methods for extreme resolution differences, and selective data persistence to eliminate redundant processing. Our approach demonstrates how distributed frameworks can be effectively adapted to handle complex scientific analysis pipelines used by domain scientists while maintaining computational efficiency.

As the global demand for sustainable food production intensifies, precision agriculture has become crucial in meeting this challenge. Traditional methods of soil data collection are often laborious and hazardous, while the complexity of processing extensive satellite data remains prohibitive for many farmers. This demonstration introduces FutureFarmNow, a novel mobile application that addresses these challenges by seamlessly integrating satellite imagery with a user-friendly mobile interface. It simplifies access to high-resolution soil data for the Colorado River Basin, enabling farmers to make informed decisions without the need for complex data processing. The application's innovative backend utilizes lightweight index facilities for efficient large-scale data analysis, bridging the gap between farmers and data scientists.

Advancements in remote sensing technology allowed for collecting vast amounts of satellite and aerial imagery with up to 1 cm pixel resolutions, stored in raster format crucial for various research fields. However, processing this data poses challenges, including resolving data dependencies when location, resolution, and coordinate systems do not align and managing large datasets within memory constraints. This paper introduces RDPro, a novel Spark-based system that efficiently processes and analyzes large raster datasets. RDPro features a new data model tailored for data dependencies in a distributed, shared-nothing environment, complete with tools for loading and writing raster data. It also optimizes core raster operations within Spark, allowing users to integrate complex data science workflows. Comparative analysis shows RDPro outperforms existing systems by up to two orders of magnitude.

Pre-processing spatial data for machine learning applications often includes combining different datasets into a form usable by the machine learning algorithms. Spatial data is generally available in two representations, raster and vector. The best data science and machine learning applications need to combine multiple datasets of both representations which is a data and compute intensive problem. This paper proposes a formal raster-vector join operator, Raptor Join, that can bridge the gap between raster and vector data. It is modeled as a relational join operator in Spark that can be easily combined with other operators, while also offering the advantage of in-situ processing. To implement the Raptor join operator efficiently, we propose a novel Flash index that has a low memory requirement and can process the entire operation with one data scan. We run an extensive experimental evaluation on large scale satellite data with up-to a trillion pixels, and big vector data with up-to hundreds of millions of segments and billions of points, and show that the proposed method can scale to big data with up-to three orders of magnitude performance gain over baselines.

Spatial data is traditionally represented using two data models, raster and vector. Raster data refers to satellite imagery while vector data includes GPS data, Tweets, and regional boundaries. While there are many real-world applications that need to process both raster and vector data concurrently, state-of-the-art systems are limited to processing one of these two representations while converting the other one which limits their scalability. This paper draws the attention of the research community to the research problems that emerge from the concurrent processing of raster and vector data. It describes three real-world applications and explains their computation and access patterns for raster and vector data. Additionally, it runs an extensive experimental evaluation using state-of-the-art big spatial data systems with raster data of up-to a trillion pixels, and vector data with up-to hundreds of millions of edges. The results show that while most systems can analyze raster and vector concurrently, but they have limited scalability for large-scale data.