A new method for SQL query implementation in a parallel execution environment Apache Spark in a package mode was developed on the basis of Bloom Filter Cascade Application (BFCA). It includes representation of the original query in a form of a few subqueries and intermediate tables, where the Bloom filters are created and applied. Then the representation is transformed into Scala code. Theoretical justification of the developed method is provided. The theoretical estimation of the network data transmission volume reduction (shuffle) is shown on an example. The efficiency of the method is demonstrated on the example of Q3 (three tables join) and Q17 (correlated subquery) queries from the TPC-H test. The developed BFCA method is more than 2 times faster and more than 10 times better on shuffle volume (as compared to Spark SQL) on Q3 at a Scale Factor SF=500. The BFCA is 8 times faster on Q17 than Hive (Spark SQL cannot implement Q17).

This article reviews record versions processing (reconciliation). It provides an algorithm for NoSQL database record versions management. It develops a record versions reconciliation model that is simultaneously updated by multiple users. The model estimates record versions reconciliation time and the number of versions simultaneously stored in a database. Based on the modeling results a designer can provide recommendations on the maximum number of users (or applications) simultaneously working with one document (database record). This is important in case there are limitations imposed on the document reconciliation time. It describes preparation and execution of experiments in a cloud, for model adequacy analysis.

У першому розділі здійснено опис предметної області, напрями діяльності. Визначено склад функцій, що входять до бізнес-процесу на основі яких розроблено схему управління бізнес-процесом. Проведено аналіз відомих програмних систем. Здійснено аналіз вимог до програмної системи. Запропоновано архітектуру програмного додатку, що дозволить краще зрозуміти функції основних його частин. Створено та описано структурну схему, основними компонентами якої є: рівень клієнта, рівень бізнес-логіки та рівень даних. Описано функціональну структуру системи та її основних елементів – модулів обробки даних. У другому розділі здійснено опис процедур тестування та їхніх результатів, описані тест-вимоги до програмного забезпечення, а також виявлені дефекти. По результатам тестування сформовано підсумок тестування. Також в даному розділі було розкрито питання встановлення та налаштування програмного забезпечення, а також вказані вимоги, дотримання яких необхідно для користування програмою.

Analysis of the existing techniques for approximate query processing of Big Data, based on sampling, histograms and wavelets, demonstrates that wavelet-based methods can be effectively utilized for OLAP purposes due to their advantages in terms of handling multidimensional data and querying single cells as well as aggregate values from a data warehouse. At the same time the current wavelet-based methods for approximate query processing have certain deficiencies making difficult to implement them in practice. In particular, most of the techniques struggle with arbitrarily size data either imposing a restriction on a dimension length to be a multiple of a power of two, or complicating decomposition algorithms what leads to the construction time increase and difficulties with error estimations. Also, there is a lack of methods for approximate processing based on wavelets with a bounded error and a confidence interval for both single and aggregate values. Our contribution in this paper is introduction of a new wavelet method for approximate query processing which handles arbitrarily sized multidimensional datasets with minor extra calculations and provides a bounded error of the single or aggregate value reconstruction. It is demonstrated that the new method allows evaluating a confidence interval of the query error depending on a given compression ratio of a data warehouse or performing an inverse task, i.e. evaluating the required data warehouse compression ratio for a given allowable error. The introduced method was applied and verified over real epidemiological datasets to support research in finding correlations and patterns in disease spread and clinical signs correlations. It was demonstrated that the accuracy of the estimated error is acceptable for retrieving single and aggregate value, query time processing advantage depends on compression ratio and volume of the processed data.