Distributed Data Processing Platforms (e.g., Hadoop, Spark, and Flink) are widely used to store and process data in a cloud environment. These platforms distribute the storage and processing of data among the computing nodes of a cloud. The efficient use of these platforms requires users to (i) configure the cloud i.e., determine the number and type of computing nodes, and (ii) tune the configuration parameters (e.g., data replication factor) of the platform. However, both these tasks require in-depth knowledge of the cloud infrastructure and distributed data processing platforms. Therefore, in this paper, we first study the relationship between the configuration of the cloud and the configuration of distributed data processing platforms to determine how cloud configuration impacts platform configuration. After understanding the impacts, we propose a co-tuning approach for recommending optimal co-configuration of cloud and distributed data processing platforms. The proposed approach utilizes machine learning and optimization techniques to maximize the performance of the distributed data processing system deployed on the cloud. We evaluated our approach for Hadoop, Spark, and Flink in a cluster deployed on the OpenStack cloud. We used various benchmarking workloads in our evaluation. Our results reveal that, in comparison to default settings, our co-tuning approach reduces execution time by 17.5% and ${\$}$ cost by 14.9% solely via configuration tuning.

The use of big data systems has become prevalent across sensitive domains, including health, defense and finance, among others. These big data systems are often complex and with complexity often comes vulnerabilities. Most big data systems also have Application Programming Interfaces (API) where these vulnerabilities can be present. This results in introducing security risks into the projects using these big data tools. To address this issue, this paper presents and uses a research process that can be used to assess the state of big data security in open source projects. The study examines tool versioning in various open source projects to uncover the vulnerabilities present and then assess the significant factors of these vulnerabilities. Furthermore, we provide insights into the vulnerabilities of different big data system APIs and how each of the vulnerabilities can manifest in a project using the specific tool. This study serves as a guideline for big data system developers to develop highly secure data-intensive systems. CCS CONCEPTS • Security and privacy → Database and storage security; Software and application security;