Data pipeline engineering plays a critical role in managing cost optimization, scalability, reliability, and data governance challenges by ensuring that big data can be ingested, processed, and stored in a scalable and reliable way, to enable analysis and insights. Addressing these obstacles necessitates a combination of human expertise, suitable technologies, and custom hardware resources. In this paper we present a survey of existing research on data pipeline engineering. Subsequently, we introduce a complex data pipeline designed to infer aircraft flights from crowdsourced networks. We implement the data pipeline using suitable big data frameworks Apache Spark and Apache Sedona. We also provide a comprehensive review of the capabilities of prominent Large Language Models (LLMs) in assisting data pipeline engineering.

There are very simple tools available that facilitate the management of email lists, such as web sites storing all emails, or e-mail digest received and stored in IMAP servers of the subscribers. These tools do not allow advanced free-text search queries and analytical queries over e-mailing list(s). In this paper, we investigate the capacity of elasticsearch -a document-oriented store, which implements inverted indexes for storing multiple collections of e-mailing lists. We propose a real-time free-text search workload as well as an analytical workload. For that purpose, we use elastic stack to implement the different tiers of the system including the data pipelines with logstash, the document-oriented store on elasticsearch, and the dashboard with Kibana.

Big data systems are becoming mainstream for big data management either for batch processing or real-time processing. In order to extract insights from data, quality issues are very important to address, particularly. A veracity assessment model is consequently needed. In this paper, we propose a model which ties quality of datasets and quality of query resultsets. We particularly examine quality issues raised by a given dataset, order attributes along their fitness for use and correlate veracity metrics to business queries. We validate our work using the open dataset NYC taxi’ trips.

The adoption and deployment of the Internet of Things (IoT) technologies in agroecology raise a challenging research agenda. Agroecology IoT projects feature complex requirements involving: heterogeneous hardware and heterogeneous software systems; data collection architectures, stream and queuing systems as well as data management systems for real-time and batch processing with different data models. On top of that, agroecology IoT applications are characterized by complex spatio-temporal data and low quality communication networks.

International audience

In this paper, inspired by the well-known Lambda architecture, we introduce a novel approach for effectively and efficiently supporting data warehouse maintenance processes in the context of near real-time OLAP scenarios, making use of so-called big summary data, and we provide an empirical study that stresses the complexity of such OLAP scenarios via using the popular TPC-H benchmark.

The purpose of this paper is twofold: First, to survey the literature on Multidimensional Database Modeling fundamentals, and Second, to sketch a research agenda which challenges Big Data four V's, namely Volume, Velocity, Veracity, and Variety.

Following our previous research results, in this paper we provide two authoritative application scenarios that build on top of OLAP*, a middleware for parallel processing of OLAP queries that truly realizes effective and efficiently OLAP over Big Data. We have provided two authoritative case studies, namely parallel OLAP data cube processing and virtual OLAP data cube design, for which we also propose a comprehensive performance evaluation and analysis. Derived analysis clearly confirms the benefits of our proposed framework.

Inspired by the emerging Big Data challenge, in this paper we provide the description of OLAP*, a Cloud based framework for supporting effective and efficient OLAP in Big Data environments. OLAP* combines data warehouse partitioning techniques with Cloud Computing paradigms, and provides a suitable implementation on top of the well-known ROLAP server Mondrian where the main task consists in applying meaningful transformation of multidimensional database schemas. We complement our analytical contribution by mean of a case study showing the effectiveness of our framework in a practical setting.

The most common benchmarks for cloud computing are the Terasort benchmark and the YCSB benchmark. Although these benchmarks are quite useful, they were not designed for data warehouse systems and related OLAP technologies. In this paper, first, we present cloud computing and data warehouse systems. Then, we argue that TPC-H benchmark - the most prominent benchmark for decision support system, mismatches cloud rationale (scalability, elasticity, pay-per-use, fault-tolerance features) and Customer Relationship Management rationale (enduser satisfaction, Quality of Service features). Finally, we present new requirements for implementing a benchmark for data warehouse systems in the cloud. The proposed requirements should allow a fair comparison of different cloud systems providers' offerings.

Our demo presents HDDBRS, a middle tier offering to clients a highly available distributed database interface using Reed Solomon codes to compute parity data. Parity data is stored in dedicated parity DB backends, is synchronously updated and allows recovering from multiple DB backend unavailability. HDDBRS middle tier is implemented in JAVA using standard technology, and is designed to be interoperable with any database engine that provides a JDBC driver and implements X/open XA protocol.

LH* RS is a high-availability scalable distributed data structure (SDDS). An LH* RS file is hash partitioned over the distributed RAM of a multicomputer, for example, a network of PCs, and supports the unavailability of any k ≥ 1 of its server nodes. The value of k transparently grows with the file to offset the reliability decline. Only the number of the storage nodes potentially limits the file growth. The high-availability management uses a novel parity calculus that we have developed, based on Reed-Salomon erasure correcting coding. The resulting parity storage overhead is about the lowest possible. The parity encoding and decoding are faster than for any other candidate coding we are aware of. We present our scheme and its performance analysis, including experiments with a prototype implementation on Wintel PCs. The capabilities of LH* RS offer new perspectives to data intensive applications, including the emerging ones of grids and of P2P computing.