The amount and diversity of data that organizations have to deal with, intensify the importance and challenges associated with data management. In this context, data catalogs play a significant role, as they are used as a tool to find and understand data. With the emergence of new approaches to storing and exploring data, such as Data Lake or Data Lakehouse, the requirements associated with building and maintaining the data catalog have evolved and represent an opportunity for organizations to develop their decision-making processes. This work explores a metric data catalog for analytical systems to support building, validating, and maintaining dashboards in a Business Intelligence system.

Abstract Developing analytical systems imposes several challenges related not only to the amount and heterogeneity of the involved data but also to the constant need to readapt and evolve to overcome new business challenges. Data are a determinant factor in the success of analytical and decision‐making applications, being its nature, availability, and quality, crucial aspects for planning and structuring populating analytical systems. Today's users are more demanding, requiring adaptable and flexible analytical applications, which impose serious challenges on extract‐transform‐load (ETL) systems design and development for ensuring flexible and robust data populating services, operating 24/7, and managing and processing large volumes of data. Thus, we should design and implement ETL processes using innovative and up‐to‐date approaches, having real application evidence. In this paper, we present a service‐oriented implementation for ETL design and development. We mapped and implemented some of the most conventional ETL processes in a service‐oriented architecture, to demonstrate the application and benefits that this kind of approach will provide to ETL systems project development.

One of the most important parts of a Data Warehousing System is the Extract-Transform-Load (ETL) component. It is responsible for extracting, transforming, conciliating, and loading data for supporting decision-making requirements. Usually, due to the complexity of managing heterogeneous data, this component is responsible for consuming most of the resources required for implementing a Data Warehousing System, representing a critical component that compromises the adequacy of the system. Despite their importance, the ETL development method is essentially ad-hoc, which does not always follow or embodies the best practices. With the emergence of Big Data and associated tools, script-based ETL became, even more, a common approach. In the last years, BPMN - Business Process Model and Notation - have been proposed and used to support ETL conceptual models. Still, as an expressive language, it provides different approaches for representing the same requirements. In this paper, we explore the use of BPMN for ETL conceptual modelling, analyzing existing approaches, and proposing a set of guidelines for using this notation in a more consistent way.

The ETL processes can be seen as typical data-oriented workflows composed of dozens of granular tasks that are responsible for the integration of data coming from different data sources. They are one of the most important components of a data warehousing system, strongly influenced by the complexity of business requirements, their changing, and evolution. To facilitate the planning and ETL implementation, a set of patterns specially designed to map standard ETL procedures is presented. They provide a simpler and conceptual perspective that can enrich to enable the generation of execution primitives. Generic models can be built, simplifying process views and providing methods for carrying out the acquired expertise to new applications using well-proven practices. This work demonstrates the fundaments of an ETL pattern-based approach for ETL development, its configuration and validation trough a set of Alloy specifications used to express its structural constraints and behaviour.

Abstract During the last few years, many research efforts have been done to improve the design of extract, transform, and load (ETL) models systems. Still, ETL systems are considered very time‐consuming, error‐prone, and complex involving several participants from different knowledge domains. The ETL processes are one of the most important components of a data warehousing system that are strongly influenced by the complexity of business requirements, their changing and evolution. These aspects influence not only the structure of the data warehouse itself but also the structures of the data sources involved with. To minimize the negative impact of such variables, we propose the use of ETL patterns to build specific ETL packages. In this paper, we formalize this approach using the BPMN for modeling more conceptual ETL workflows, mapping them to real execution primitives through the use of a domain‐specific language that allows for the generation of specific instances that can be executed in an ETL commercial tool.

The use of software patterns is a common practice in software design, providing reusable solutions for recurring problems. Patterns represent a general skeleton used to solve common problems, providing a way to share regular practices and reduce the resources needed for implementing software systems. Data warehousing populating processes are a very particular type of software used to migrate data from one or more data sources to a specific data schema used to support decision support activities. The quality of such processes should be guarantee. Otherwise, the final system will deal with data inconsistencies and errors, compromising its suitability to support strategic business decisions. To minimize such problems, we propose a pattern-oriented approach to support ETL lifecycle, from conceptual representation to its execution primitives using a specific commercial tool. An ontology-based meta model it was designed and used for describing patterns internal specification and providing the means to support and enable its configuration and instantiation using a domain specific language.

Usually, data warehousing populating processes are data-oriented workflows composed by dozens of granular tasks that are responsible for the integration of data coming from different data sources. Specific subset of these tasks can be grouped on a collection together with their relationships in order to form higher- level constructs. Increasing task granularity allows for the generalization of processes, simplifying their views and providing methods to carry out expertise to new applications. Well-proven practices can be used to describe general solutions that use basic skeletons configured and instantiated according to a set of specific integration requirements. Patterns can be applied to ETL processes aiming to simplify not only a possible conceptual representation but also to reduce the gap that often exists between two design perspectives. In this paper, we demonstrate the feasibility and effectiveness of an ETL pattern-based approach using task clustering, analyzing a real world ETL scenario through the definitions of two commonly used clusters of tasks: a data lookup cluster and a data conciliation and integration cluster.

ETL processes are difficult components of modern Data Warehousing architectures, which are gaining momentum thanks the renewed interest stirred-up by the emerging Big Data paradigm. Due to the complexity of modeling ETL processes, various alternatives appeared in the literature. But several limitations arise, such as the poor attention to the flexibility of models. Following the drawbacks of actual proposals, in this paper we argue that a novel approach based on a pattern-oriented and task-reusable framework may convey in lot of advantages for the ETL process (conceptual) modeling itself. Our proposal is thus represented by a pattern-oriented meta-model for supporting the conceptual modeling of ETL processes in next-generation Data Warehousing environments. The most distinctive characteristic of our framework consists in the reusability of task-oriented components, with clear advantages for the flexibility and the extendibility of the final ETL process model so-determined. To validate our proposal, we exploit the YAWL workflow modeling language, and develop a suitable pattern for validating target ETL processes.

Due to their complexity and importance on data warehousing systems development, ETL systems are being a strong research subject approaching several issues especially related to reduce implementation costs, and enhance decision-making processes abilities. Since ETL systems are intrinsically connected to business processes, they are strongly influenced by requirements changing, which impact ranges from changes in data source structures to the integration of new business rules and requirements. In this paper, we propose the use of ETL patterns not only to specify ETL systems internal behaviour but also to describe how patterns communicate and interchange data among them. In order to test and demonstrate the feasibility of our approach, we used the Reo coordination language for conceptual modelling, which provided an understandable notation and methodology with the ability to reduce implementation errors and development costs.

The formalization of software patterns has proven to be very useful in software developing, improving systems communication, data interchange across platforms, and simplifying the integration of processes and data flows. Populating a data warehouse (ETL) is often a very complex task demanding significant computational resources. It faces many drawbacks during its design and implementation, involving not only large volumes of data that must be processed but also undesirable change of business requirements. All of this leads frequently to reuse significant parts of other ETL implementations, adapting data structures and processes to comply with new requirements. Additionally, we believe that it's necessary a more simply and reliable approach for ETL conceptual modelling covering the "lack of mature" of this important part of ETL development. In this paper we explored a new approach to ETL conceptual modelling using the Reo coordination language, trying to evaluate its adequacy and expressiveness on the coordination of ETL tasks. A pattern-based approach was designed to map typical operations used in real world ETL scenarios from an initial Reo specification. For demonstration purposes, we present and discuss as two case studies, a slowly changing dimension and a surrogated key pipelining processes.