Many of the ETL products in the market today provide tools for design of ETL workflows, with very little or no support for opti-mization of such workflows. Optimization of ETL workflows pose several new challenges compared to traditional query optimization in database systems. There have been many attempts both in the industry and the research community to support cost-based opti-mization techniques for ETL Workflows, but with limited success. Non-availability of source statistics in ETL is one of the major chal-lenges that precludes the use of a cost based optimization strategy. However, the basic philosophy of ETL workflows of design once and execute repeatedly allows interesting possibilities for determin-ing the statistics of the input. In this paper, we propose a frame-work to determine various sets of statistics to collect for a given workflow, using which the optimizer can estimate the cost of any alternative plan for the workflow. The initial few runs of the work-flow are used to collect the statistics and future runs are optimized based on the learned statistics. Since there can be several alterna-tive sets of statistics that are sufficient, we propose an optimization framework to choose a set of statistics that can be measured with the least overhead. We experimentally demonstrate the effective-ness and efficiency of the proposed algorithms.

Master data management (MDM) integrates data from multiple structured data sources and builds a consolidated 360-degree view of business entities such as customers and products. Today's MDM systems are not prepared to integrate information from unstructured data sources, such as news reports, emails, call-center transcripts, and chat logs. However, those unstructured data sources may contain valuable information about the same entities known to MDM from the structured data sources. Integrating information from unstructured data into MDM is challenging as textual references to existing MDM entities are often incomplete and imprecise and the additional entity information extracted from text should not impact the trustworthiness of MDM data. In this paper, we present an architecture for making MDM text-aware and showcase its implementation as IBM Info-Sphere MDM Extension for Unstructured Text Correlation, an add-on to IBM InfoSphere Master Data Management Standard Edition. We highlight how MDM benefits from additional evidence found in documents when doing entity resolution and relationship discovery. We experimentally demonstrate the feasibility of integrating information from unstructured data sources into MDM.

Most modern RDBMS depend on the query processing optimizer's cost model to choose the best execution plan for a given query. Since the physical IO (PIO) is a costly operation to execute, it naturally has an important weight in RDBMS classical cost models, which assume that the data is disk-resident and does not fit in the available main memory. However, this assumption is no longer true with the advent of cheap large main memories.In this paper, we discuss the importance of considering the buffer-cache occupancy during optimization and propose the Warm Cache Costing (WCC) model as a new technique for buffer-pool aware query optimization. The WCC-model is a novel feedback optimization technique, based on the execution statistics by learning PIO-compensation (PIOC) factors. The PIOC factor defines the average percentage of a table which is cached in the buffer pool. These PIOC factors are used in subsequent query optimizations to better estimate the PIO, thus leading to better plans.These techniques have been implemented in Sybase Adaptive Server Enterprise (ASE) database system. We have observed that they provide considerable improvements in query timings, in Decision Support environments and with almost negligible regression(if any) in other environments. This model enjoys the advantage of requiring no change to the buffer manager or other modules underlying the query processing layer and therefore is easy to implement. Also, since this model is part of an extensive feedback optimization architecture, other techniques using feedback optimization framework can be plugged in easily.