Performance monitoring systems collect and analyze data, and provide insight on availability, performance, and capacity from various monitored targets. However, the analytics in online performance monitoring systems have often been limited due to the query performance of large-scale multidimensional data. The existing approaches for preference or ranking queries generally include approaches with and without pre-processing. The approaches without pre-processing work well with small data sets. For larger datasets, the preference query is usually slow or not time feasible. In this paper, we introduce and evaluate an optimized query approach for performance monitoring systems using the bit-sliced index (BSI). The objective is to enable users to perform fast interactive queries with various querying criteria on multiple dimensions. The experiments cover preference top-k queries on the proposed approach with BSI, an approach using bitmap indexing for top-k queries, and the sequential scan sort top-k selection algorithms. The evaluation covers the single attribute query, multiple attributes weighted sum query, and multidimensional grouping using a real-time performance monitoring system data with several hundred thousand records. Working with a single attribute top-k evaluation, the proposed bit-sliced approach on average outperforms the bitmap approach and the sequential scan approach by a factor of 2 and a factor of 100+ respectively. Similarly, with multiple attributes sum top-k evaluation, it is 5 times faster than the sequential scan approach as well.

fraction of the points while a constant penalty is applied for the rest of the points. QED not only improves the quality of the distance metric, but also improves query time performance by filtering out non relevant data. We propose a distributed indexing and query algorithm to efficiently compute QED. Our experimental results show improvements in classification accuracy as well as query performance up to one order of magnitude faster than Manhattan-based sequential scan NN queries over datasets with hundreds of dimensions.

We present an empirical performance evaluation of algorithms that replace arithmetic operations with low-level bit operations for power-aware Big Data processing. Specifically, we compare two different data structures in terms of both execution time and power efficiency: (a) a baseline design using arrays, and (b) a design using bit-slice indexing (BSI) and distributed BSI arithmetic. We evaluate two types of queries popular in Big Data analytics: aggregations and top-k. These queries were implemented using each of the two data structure designs on Apache Spark running on a server cluster that was instrumented with specialized hardware for synchronized real-time power measurement for each server in the cluster. We performed a series of experiments running the above queries on several different datasets. These experiments show that the bit-slicing algorithm consistently outperforms the array algorithm in both power efficiency and execution time. An interesting observation is that the power efficiency improvement of the bit-slicing algorithm over the array method is comparable to or greater than the improvement in execution time for both queries evaluated.

The advent of satellite imagery, remote sensing products, and global scale numerical climate models over the last two decades has created an explosion of available gridded environmental data. These space-time explicit datasets are produced and distributed using different spatial and temporal resolutions. Current approaches for comparing two different products generally involve offline pre-computation of aggregations to a common spatio-temporal resolution. This limits the user's ability to interactively compare different data products or transform data products into the required input resolution for modeling.

Preference (top-k) queries play a key role in modern data analytics tasks. Top-k techniques rely on ranking functions in order to determine an overall score for each of the objects across all the relevant attributes being examined. This ranking function is provided by the user at query time, or generated for a particular user by a personalized search engine which prevents the pre-computation of the global scores. Executing this type of queries is particularly challenging for high-dimensional data. Recently, bit-sliced indices (BSI) were proposed to answer these high-dimensional preference queries efficiently in a centralized environment.

Preference (top-k) queries play a key role in modern data analytics tasks. Top-k techniques rely on ranking functions in order to determine an overall score for each of the objects across all the relevant attributes being examined. This ranking function is provided by the user at query time, or generated for a particular user by a personalized search engine which prevents the pre-computation of the global scores. Executing this type of queries is particularly challenging for high-dimensional data. Recently, bit-sliced indices (BSI) were proposed to answer these preference queries efficiently in a non-distributed environment for data with hundreds of dimensions. As MapReduce and key-value stores proliferate as the preferred methods for analyzing big data, we set up to evaluate the performance of BSI in a distributed environment, in terms of index size, network traffic, and execution time of preference (top-k) queries, over data with thousands of dimensions. Indexing is implemented on top of Apache Spark for both column and row stores and shown to outperform Hive when running on Map-reduce, and Tez for top-k (preference) queries.

Bitmap indices are widely used for large read-only repositories in data warehouses and scientific databases. Their binary representation allows for the use of bitwise operations and specialized run-length compression techniques. Due to a trade-off between compression and query efficiency, bitmap compression schemes are aligned using a fixed encoding length size (typically the word length) to avoid explicit decompression during query time. In general, smaller encoding lengths provide better compression, but require more decoding during query execution. However, when the difference in size is considerable, it is possible for smaller encodings to also provide better execution time. We posit that a tailored encoding length for each bit vector will provide better performance than a one-size-fits-all approach. We present a framework that optimizes compression and query efficiency by allowing bitmaps to be compressed using variable encoding lengths while still maintaining alignment to avoid explicit decompression. Efficient algorithms are introduced to process queries over bitmaps compressed using different encoding lengths. An input parameter controls the aggressiveness of the compression providing the user with the ability to tune the tradeoff between space and query time. Our empirical study shows this approach achieves significant improvements in terms of both query time and compression ratio for synthetic and real data sets. Compared to 32-bit WAH, VAL-WAH produces up to 1.8× smaller bitmaps and achieves query times that are 30% faster.

Many large-scale read-only databases and data warehouses use bitmap indices in an effort to speed up data analysis. These indices have the dual properties of compressibility and being able to leverage fast bit-wise operations for query processing. Numerous hybrid run-length encoding compression schemes have been proposed that greatly compress the index and enable querying without the need to decompress. Typically, these schemes align their compression with the computer architecture's word size to further accelerate queries.