Scheduling and processing decisions in multiplatform data systems can lead to efficient execution of workflow tasks across available platforms. In this work, we discuss how to create a cross-platform system for Data Science tasks that leverages modern data abstractions, such as visual programming and DataFrames. Our approach uses an extended graph to represent workflows, capturing various execution aspects, including dataset evolution, execution costs, and data migration costs across different platforms and environments. This graph can be further transformed based on predefined rules, optimization techniques, and the application of machine learning tools to guide decision-making during the scheduling of the blocks that compose the workflows in question. Empirically, we demonstrate that our prototype, which initially supports Spark and Pandas, can identify not only the best platform with 90.8% accuracy, or the best execution environment for an entire workflow (83.8%), but also achieve performance gains of up to 2.5 times by scheduling tasks on the best platform for each activity.

Record Deduplication (RD) aims to identify instances that represent the same real-world entity in data repositories. In the government environment, the RD process facilitates the identification of irregularities and reduces the consumption of computing resources in data integration tasks. In this context, we propose a scalable, effective and efficient platform, called DedupeGov, for integrating large data repositories (i.e., with large volumes of data, in the order of millions of records) to unify duplicate entities from multiple and different sources. Our experimental results indicate a 21.8% of reduction in the number of records of the original repository with 99% of precision and 95% of recall when identifying duplicate records. In addition, our platform was capable of building more complete records, eliminating at least 32% of records with null attributes. Furthermore, our solution is very efficient and scalable for large volumes of data, deduplicating a repository of almost 400 million records in around one hour, besides being easy to generalize to different types of entity.