We study the problem of optimizing data storage and access costs on the cloud while ensuring that the desired performance or latency is unaffected. We first propose an optimizer that optimizes the data placement tier (on the cloud) and the choice of compression schemes to apply, for given data partitions with temporal access predictions. Secondly, we propose a model to learn the compression performance of multiple algorithms across data partitions in different formats to generate compression performance predictions on the fly, as inputs to the optimizer. Thirdly, we propose to approach the data partitioning problem fundamentally differently than the current default in most data lakes where partitioning is in the form of ingestion batches. We propose access pattern aware data partitioning and formulate an optimization problem that optimizes the size and reading costs of partitions subject to access patterns.We study the various optimization problems theoretically as well as empirically, and provide theoretical bounds as well as hardness results. We propose a unified pipeline of cost minimization, called SCOPe that combines the different modules. We extensively compare the performance of our methods with related baselines from the literature on TPC-H data as well as enterprise datasets (ranging from GB to PB in volume) and show that SCOPe substantially improves over the baselines. We show significant cost savings compared to platform baselines, of the order of 50% to 83% on enterprise Data Lake datasets that range from terabytes to petabytes in volume.

Context: Systematic literature reviews and mapping studies usually rely on automated searches of digital libraries to identify primary studies. Defining proper search strings, executing semantically similar searches on different libraries, and reporting limitations of searches increase the reliability of secondary studies. Objective: We aim to survey the current state of using automated searches in secondary software engineering studies. In particular, we aim at analyzing how automated searches are reported and at understanding the reproducibility of secondary studies. Method: We perform a preliminary tertiary study that covers 50 recently published representative secondary studies from different software engineering venues and subfields. Results: We found that most secondary studies complement an automated search with a manual search and use four or more digital libraries. Also, we found that the quality of reporting search strings is rather poor. Finally, we found that most secondary studies do not acknowledge limitations of automated searches and implications of limitations on study findings. Conclusions: Our findings highlight implications for researchers (e.g., to properly report the search process) and for reviewers (e.g., to execute search strings reported in papers). Also, our findings indicate that secondary studies are difficult to replicate.

Search phase is considered as one of the most important steps in conducting secondary studies such as systematic literature reviews and mapping studies. In recent times, automatic search in digital libraries and academic search engines has been the preferred method of search phase execution for most software engineering related secondary studies. However, there are no previous studies that report or evaluate the secondary study relevant search features of these electronic data sources. We perform a feature analysis (screening mode) based evaluation of five widely used digital libraries (IEEE Xplore, ACM DL, SpringerLink, ScienceDirect and Wiley) in terms of their respective features required to support the search phase of secondary studies. We identify a total of 68 search related features and conduct a comprehensive exploration into their execution behaviors. The overall work presents a useful caution guide for systematic reviewers who plan to use the identified features for executing various search phase steps of their secondary studies.