Demand for enterprise data warehouse solutions to support real-time Online Transaction Processing (OLTP) queries as well as long-running Online Analytical Processing (OLAP) workloads is growing. Greenplum database is traditionally known as an OLAP data warehouse system with limited ability to process OLTP workloads. In this paper, we augment Greenplum into a hybrid system to serve both OLTP and OLAP workloads. The challenge we address here is to achieve this goal while maintaining the ACID properties with minimal performance overhead. In this effort, we identify the engineering and performance bottlenecks such as the under-performing restrictive locking and the two-phase commit protocol. Next we solve the resource contention issues between transactional and analytical queries. We propose a global deadlock detector to increase the concurrency of query processing. When transactions that update data are guaranteed to reside on exactly one segment we introduce one-phase commit to speed up query processing. Our resource group model introduces the capability to separate OLAP and OLTP workloads into more suitable query processing mode. Our experimental evaluation on the TPC-B and CH-benCHmark benchmarks demonstrates the effectiveness of our approach in boosting the OLTP performance without sacrificing the OLAP performance.

© 2016, Copyright is with the authors. Although existing database systems provide users an efficient means to select tuples based on attribute criteria, they however provide little means to select tuples based on whether they meet aggregate requirements. For instance, a requirement may be that the cardinality of the query result must be 1000 or the sum of a particular attribute must be < $5000. In this work, we term such queries as "Aggregation Constrained Queries" (ACQs). Aggregation constrained queries are crucial in many decision support applications to maintain a product's competitive edge in this fast moving field of data processing. The challenge in processing ACQs is the unfamiliarity of the underlying data that results in queries being either too strict or too broad. Due to the lack of support of ACQs, users have to resort to a frustrating trial-and-error query refinement process. In this paper, we introduce and define the semantics of ACQs. We propose a refinement-based approach, called ACQUIRE, to efficiently process a range of ACQs. Lastly, in our experimental analysis we demonstrate the superiority of our technique over extensions of existing algorithms. More specifically, ACQUIRE runs up to 2 orders of magnitude faster than compared techniques while producing a 2X reduction in the amount of refinement made to the input queries.

Public vehicles and personal navigation assistants have become increasingly equipped with single-frequency global positioning system (GPS) receivers or loggers. These commonly used terminals offer an inexpensive way for acquiring large volumes of GPS traces, which contain information pertaining to road position and traffic rules. Using this new type of spatial data resource, we propose a novel approach for generating high-quality routable road maps. In this approach, a simplified road network graph model uses circular boundaries to separate all GPS traces into road intersections and road segments and builds road networks that maintain their identical geometric topologies through the entry/exit points at the original boundaries. One difficulty inherent to this type of approach is how to best determine the appropriate spatial coverage for road intersections. Conflict points among GPS traces that have large intersection angles usually occur within the physical areas of road intersections, particularly those involving left turns. Therefore, we determined a proper circle boundary for individual road intersections by conducting a spatial analysis of such feature points. This approach was implemented using Python and PostgreSQL/PostGIS and was tested in Huaibei City, China. Based on a comparison with human-interpreted results, the automatically generated routable road map was demonstrated to be of high quality and displayed detailed road networks with turning at various at-grade intersections, interchanges and U-turns.

Big Data analytics often include complex queries with similar or identical expressions, usually referred to as Common Table Expressions (CTEs). CTEs may be explicitly defined by users to simplify query formulations, or implicitly included in queries generated by business intelligence tools, financial applications and decision support systems. In Massively Parallel Processing (MPP) database systems, CTEs pose new challenges due to the distributed nature of query processing, the overwhelming volume of underlying data and the scalability criteria that systems are required to meet. In these settings, the effective optimization and efficient execution of CTEs are crucial for the timely processing of analytical queries over Big Data. In this paper, we present a comprehensive framework for the representation, optimization and execution of CTEs in the context ofOrca-- Pivotal's query optimizer for Big Data. We demonstrate experimentally the benefits of our techniques using industry standard decision support benchmark.

Real-time analytical systems need to handle workloads comprised of expensive decision support queries with diverse quality of ser-vice requirements known as contracts. Contract driven multi-query processing, being an NP-hard problem, remains unaddressed to date. The traditional approach of blindly pipelining the entire input through a shared execution plan is not viable due to the diversity in query contracts. To tackle this challenge, we now develop a flexi-ble model to express contracts and accompany it with an effective means to measure the run-time contract satisfaction. We propose our Contract-Aware Query Execution framework CAQE. In this work, we exploit the principle that “different portions of the in-put contribute to disparate subsets of queries with varying degrees of progressiveness. ” Therefore, CAQE’s processing of the input chunks is driven by how the different query contracts are being met at run-time. To maximize the contract satisfaction of the work-load, CAQE leverages the dependencies of input chunks across the queries. This enables us to determine the impact of processing par-ticular input chunks on improving the run-time contract satisfac-tion. Our experiments demonstrate the effectiveness of CAQE in increasing the overall contract satisfaction of the workload, specifi-cally 2 fold better than existing multi-query processing techniques. 1.

The performance of analytical query processing in data management systems depends primarily on the capabilities of the system's query optimizer. Increased data volumes and heightened interest in processing complex analytical queries have prompted Pivotal to build a new query optimizer.

This study developed a method for Soil Organic Matter (SOM) mapping in a lithoidal mountainous area using a stratified strategy under fuzzy inference framework. Environmental variables derived from terrain analysis and remote sensing incorporated with a small number of field samples were used to predict spatial variation of SOM. A new variable, bare soil ratio, representing the fractions of bare soil area inside a pixel of remote sensing images resulted from sub-pixel unmixing technology is used specifically for rocky upland area. A case study in the Hushiha area of North China has shown an improved spatial detail and accuracy of SOM compared to the available traditional soil map of Heibei Province. The proposed method takes full advantages of data with free access such as SRTM DEM and remote sensing images of HJ-A which exhibits massive potential to map SOM efficiently at low cost for a large area.

This paper presents a weighted clustering algorithm based on the physical attraction model, which improves the physical attraction model by assigning a different weight to the position points on a GPS trace for a fast convergence according to their velocity and directional changes. The physical attraction model pulls together traces that belong on the same road in response to simulated potential energy wells created around each trace. Assuming a vehicle with a high velocity has a little derivation to the road it runs on, we assign a high weight to its trace in the physical attraction model so that the clustering progress converges rapidly and closely to road. Within the clustering process, an angle-threshold based smoothing filter is appended to keep the consistency between the changes of those points on the adjusted trace at each loop. This algorithm was demonstrated to enable to effectively clarify those vehicle GPS traces on the same road. In comparison of previous work, it also shows an improved quality grouping GPS traces near road-crossing area by embedding the smoothing filter in clustering process.

Multi-criteria decision support (MCDS) is crucial in many business and web applications such as web searches, B2B portals and on-line commerce. Such MCDS applications need to report results early; as soon as they are being generated so that they can react and formulate competitive decisions in near real-time. The ease in expressing user preferences in web-based applications has made Pareto-optimal (skyline) queries a popular class of MCDS queries. However, state-of-the-art techniques either focus on handling skylines on single input sets (i.e., no joins) or do not tackle the challenge of producing progressive early output results. In this work, we propose a progressive query evaluation framework ProgXe that transforms the execution of queries involving skyline over joins to be non-blocking, i.e., to be progressively generating results early and often. In ProgXe the query processing (join, mapping and skyline) is conducted at multiple levels of abstraction, thereby exploiting the knowledge gained from both input as well as mapped output spaces. This knowledge enables us to identify and reason about abstract-level relationships to guarantee correctness of early output. It also provides optimization opportunities previously missed by current techniques. To further optimize ProgXe, we incorporate an ordering technique that optimizes the rate at which results are reported by translating the optimization of tuple-level processing into a job-sequencing problem. Our experimental study over a wide variety of data sets demonstrates the superiority of our approach over state-of-the-art techniques.

We demonstrate ProgXe, a practical approach to support Multi-Criteria Decision Support (MCDS) applications that need to report results as they are being generated to enable the user to make competitive decisions. ProgXe transforms the execution of MCDS queries involving skyline over joins to be non-blocking by progressively generating results early and often. The demonstration highlights key features of our progressive execution framework that optimizes for early output generation by: (1) evaluating the query at multiple levels of abstraction, (2) exploiting the skyline knowledge gained from both input as well as mapped output spaces. The audience will be able to submit MCDS queries. We provide visualization tools that enable the user to make quick decisions, compare alternative techniques, and provide capability to fine-tune the query predicates based on the early output results.

In many business and consumer applications, queries have cardinality constraints. However, current database systems provide minimal support for cardinality assurance. Consequently, users must adopt a cumbersome trial-and-error approach to find queries that are close to the original query but also attain the desired cardinality. In this demonstration, we present QRelX a novel framework to automatically generate alternate queries that meet the cardinality and closeness criteria. QRelX employs an innovative query space transformation strategy, proximity-based search and incremental cardinality estimation to efficiently find alternate queries. Our demonstration is an interactive game that allows the audience to compete with QRelX via manual query refinement. We illustrate the importance of cardinality assurance through real-time comparisons between manual refinement and QRelX. We also highlight the novelty of our solution by visualizing the core algorithms of QRelX.

Continuous time-series sequence matching, specifically, matching a numeric live stream against a set of redefined pattern sequences, is critical for domains ranging from fire spread tracking to network traffic monitoring. While several algorithms exist for similarity matching of static time-series data, matching continuous data poses new, largely unsolved challenges including online real-time processing requirements and system resource limitations for handling infinite streams. In this work, we propose a novel live stream matching framework, called n-Snippet Indices Framework (in short, SNIF), to tackle these challenges. SNIF employs snippets as the basic unit for matching streaming time-series. The insight is to perform the matching at two levels of granularity: bag matching of subsets of snippets of the live stream against prefixes of the patterns, and order checking for maintaining successive candidate snippet bag matches. We design a two-level index structure, called SNIF index, which supports these two modes of matching. We propose a family of online two-level prefix matching algorithms that trade off between result accuracy and response time. The effectiveness of SNIF to detect patterns has been thoroughly tested through experiments using real datasets from the domains of fire monitoring and sensor motes. In this paper, we also present a study of SNIF's performance, accuracy and tolerance to noise compared against those of the state-of-the-art Continuous Query with Prediction (CQP) approach.

This demonstration presents FireStream, a sensor stream processing system which provides services for run-time detection, monitoring and visualization of fire spread in intelligent buildings that can be of great benefit to first responders. Our system can effectively handle large heterogeneous sensor streams using shared window execution and dynamic participant handling to yield a high-ary MJoin solution.