Enabling interactive data exploration at cloud scale requires minimizing end-to-end query execution latency, while guaranteeing fault tolerance, and query execution under resource-constraints. Typically, such a query execution involves orchestrating the execution of hundreds or thousands of related tasks on cloud scale clusters. Without any resource constraints, all query tasks can be scheduled to execute simultaneously (gang scheduling) while connected tasks stream data between them. When the data size referenced by a query increases, gang scheduling may be resource-wasteful or un-satisfiable with a limited, per-query resource budget. This paper introduces B ubble E xecution , a new query processing framework for interactive workloads at cloud scale, that balances cost-based query optimization, fault tolerance, optimal resource management, and execution orchestration. Bubble execution involves dividing a query execution graph into a collection of query sub-graphs (bubbles), and scheduling them within a per-query resource budget. The query operators (tasks) inside a bubble stream data between them while fault tolerance is handled by persisting temporary results at bubble boundaries. Our implementation enhances our JetScope service, for interactive workloads, deployed in production clusters at Microsoft. Experiments with TPC-H queries show that bubble execution can reduce resource usage significantly in the presence of failures while maintaining performance competitive with gang execution.

Managed Hadoop in the cloud, especially SQL-on-Hadoop, has been gaining attention recently. On Platform-as-a-Service (PaaS), analytical services like Hive and Spark come pre-configured for general-purpose and ready to use. Thus, giving companies a quick entry and on-demand deployment of ready SQL-like solutions for their big data needs. This study evaluates cloud services from an end-user perspective, comparing providers including: Microsoft Azure, Amazon Web Services, Google Cloud, and Rackspace. The study focuses on performance, readiness, scalability, and cost-effectiveness of the different solutions at entry/test level clusters sizes. Results are based on over 15,000 Hive queries derived from the industry standard TPC-H benchmark. The study is framed within the ALOJA research project, which features an open source benchmarking and analysis platform that has been recently extended to support SQL-on-Hadoop engines. The ALOJA Project aims to lower the total cost of ownership (TCO) of big data deployments and study their performance characteristics for optimization. The study benchmarks cloud providers across a diverse range instance types, and uses input data scales from 1GB to 1TB, in order to survey the popular entry-level PaaS SQL-on-Hadoop solutions, thereby establishing a common results-base upon which subsequent research can be carried out by the project. Initial results already show the main performance trends to both hardware and software configuration, pricing, similarities and architectural differences of the evaluated PaaS solutions. Whereas some providers focus on decoupling storage and computing resources while offering network-based elastic storage, others choose to keep the local processing model from Hadoop for high performance, but reducing flexibility. Results also show the importance of application-level tuning and how keeping up-to-date hardware and software stacks can influence performance even more than replicating the on-premises model in the cloud.

During the past years the exponential growth of data, its generation speed, and its expected consumption rate presents one of the most important challenges in IT both for industry and research. For these reasons, the ALOJA research project was created by BSC and Microsoft as an open initiative to increase cost-efficiency and the general understanding of Big Data systems via automation and learning. The development of the project over its first year, has resulted in a open source benchmarking platform used to produce the largest public repository of Big Data results1, featuring over 42,000 job execution details. ALOJA also includes web-based analytic tools to evaluate and gather insights about cost-performance of benchmarked systems. The tools offer means to extract knowledge that can lead to optimize configuration and deployment options in the Cloud i.e., selecting the most cost-effective VMs and cluster sizes. This article describes the evolution of the project focus and research lines, for a period of over a year while continuously benchmarking systems for Big Data. As well discusses the motivation - both technical and market-based - of such changes. It also presents the main results from the evaluation of different OS and Hadoop configurations, covering over 100 hardware deployments. During this time, ALOJA's initial target has shifted from a previous low-level profiling of Hadoop runtime with HPC tools, passing through extensive benchmarking and evaluation of a large body of results via aggregation, to currently leveraging Predictive Analytics (PA) techniques. The ongoing efforts in PA show promising results to automatically model the behavior of systems i.e., predicting job execution times with high accuracy or to reduce the number of benchmark runs needed. As well as for Knowledge Discovery (KD) to find relations among software and hardware components. Techniques that jointly support foresighting cost-effectiveness of new defined systems, reducing benchmarking time and costs.

This article presents the ALOJA project, an initiative to produce mechanisms for an automated characterization of cost-effectiveness of Hadoop deployments and reports its initial results. ALOJA is the latest phase of a long-term collaborative engagement between BSC and Microsoft which, over the past 6 years has explored a range of different aspects of computing systems, software technologies and performance profiling. While during the last 5 years, Hadoop has become the de-facto platform for Big Data deployments, still little is understood of how the different layers of the software and hardware deployment options affects its performance. Early ALOJA results show that Hadoop's runtime performance, and therefore its price, are critically affected by relatively simple software and hardware configuration choices e.g., number of mappers, compression, or volume configuration. Project ALOJA presents a vendor-neutral repository featuring over 5000 Hadoop runs, a test bed, and tools to evaluate the cost-effectiveness of different hardware, parameter tuning, and Cloud services for Hadoop. As few organizations have the time or performance profiling expertise, we expect our growing repository will benefit Hadoop customers to meet their Big Data application needs. ALOJA seeks to provide both knowledge and an online service to with which users make better informed configuration choices for their Hadoop compute infrastructure whether this be on-premise or cloud-based. The initial version of ALOJA's Web application and sources are available at http://hadoop.bsc.es

Presents the introductory welcome message from the conference proceedings.

The integration of the .NET Common Language Runtime (CLR) into the SQL Server DBMS enables rich business logic written in modern .NET languages to run close to the data. Database application developers can write business logic as functions, stored procedures, and triggers. They can also extend the native capabilities of the DBMS by adding new scalar data types, and aggregates. A previous paper [2] described the architecture and design principles of the integration of the CLR inside SQL Server. Here we present new aspects of this work. First, we describe the extensibility contracts for user-defined types and aggregates in detail. Second, we present the advances to the CLR integration in SQL Server 2008 which significantly enhances the breath of applications supported by SQL Server. In particular, we describe the support for large (greater than 8000 byte) user-defined types and aggregates, multiple-input user-defined aggregates, and order-aware table valued functions. Third, we show how we leveraged scalar type extensibility to provide a hierarchical identifier data type that enables encoding of keys describing hierarchies as well as built-in support for spatial applications. This support includes both flat- and round-earth spatial types, as well as a spatial index. Fourth, we present how we use Language Integrated Query (LINQ) enhancements in .NET languages to improve developer productivity when creating routines that require data access. Finally, we present preliminary performance results showing the efficiency of streaming TVFs and aggregates relative to equivalent native features.

Traditional client-server applications relegate query and persistence operations on their data to database systems. The database system operates on data in the form of rows and tables, while the application operates on data in terms of higher-level programming language constructs (classes, structures etc.). The impedance mismatch in the data manipulation services between the application and the database tier was problematic even in traditional systems. With the advent of service-oriented architectures (SOA), application servers and multi-tier applications, the need for data access and manipulation services that are well-integrated with programming environments and can operate in any tier has increased tremendously. Microsoft‘s ADO.NET Entity Framework is a platform for programming against data that raises the level of abstraction from the relational level to the conceptual (entity) level, and thereby significantly reduces the impedance mismatch for applications and data-centric services. This paper describes the key aspects of the Entity Framework, the overall system architecture, and the underlying technologies. Categories and Subject Descriptors:

This paper presents an architecture overview of the distributed, heterogeneous query processor (DHQP) in the Microsoft SQL server database system to enable queries over a large collection of diverse data sources. The paper highlights three salient aspects of the architecture. First, the system introduces well-defined abstractions such as connections, commands, and rowsets that enable sources to plug into the system. These abstractions are formalized by the OLE DB data access interfaces. The generality of OLE DB and its broad industry adoption enables our system to reach a very large collection of diverse data sources ranging from personal productivity tools, to database management systems, to file system data. Second, the DHQP is built-in to the relational optimizer and execution engine of the system. This enables DH queries and updates to benefit from the cost-based algebraic transformations and execution strategies available in the system. Finally, the architecture is inherently extensible to support new data sources as they emerge as well as serves as a key extensibility point for the relational engine to add new features such as full-text search and distributed partitioned views.

The integration of the .NET Common Language Runtime (CLR) inside the SQL Server DBMS enables database programmers to write business logic in the form of functions, stored procedures, triggers, data types, and aggregates using modern programming languages such as C#, Visual Basic, C++, COBOL, and J++. This paper presents three main aspects of this work. First, it describes the architecture of the integration of the CLR inside the SQL Server database process to provide a safe, scalable, secure, and efficient environment to run user code. Second, it describes our approach to defining and enforcing extensibility contracts to allow a tight integration of types, aggregates, functions, triggers, and procedures written in modern languages with the DBMS. Finally, it presents initial performance results showing the efficiency of user-defined types and functions relative to equivalent native DBMS features.

With an explosion of data on the web, consistent data access to diverse data sources has become a challenging task. In this tutorial will present topics of interest to database researchers and developers building: interoperable middle-ware, gateways, distributed heterogeneous query processors, federated databases, data source wrappers, mediators, and DBMS extensions.

Microsoft Universal Data Access defines a platform for developing multi-tier enterprise applications that require efficient access to diverse relational or non-relational data sources across intranets or the Internet. Universal Data Access consists of a collection of software components that interact with each other using system-level interfaces defined by OLE DB and providing an application-level data access model called ActiveX Data Objects (ADO). This talk provides an overview of the platform.

The article describes an effort at Microsoft whose primary goal is to enable applications to have uniform access to data stored in diverse DBMS and non DBMS information containers. Applications continue to take advantage of the benefits of database technology such as declarative queries, transactional access, and security without having to transfer data from its place of origin to a DBMS. Our approach consists of defining an open, extensible collection of interfaces that factor and encapsulate orthogonal, independently reusable portions of DBMS functionality. These interfaces define the boundaries of DBMS components arch as record containers, and query processors that enable uniform, transactional access to data among such components. The proposed interfaces extend Microsoft's OLE Component Object Model (COM) with database functionality, hence these interfaces are collectively referred to as OLE DB. The OLE DB functional areas include data access and updates (rowsets), query processing, catalog information, notifications, transactions, security, and distribution. The article presents an overview of the OLE DB approach and its areas of componentization.

This paper presents an overview of OLE DB, a set of interfaces being developed at Microsoft whose goal is to enable applications to have uniform access to data stored in DBMS and non-DBMS information containers. Applications will be able to take advantage of the benefits of database technology without having to transfer data from its place of origin to a DBMS. Our approach consists of defining an open, extensible Collection of interfaces that factor and encapsulate orthogonal, reusable portions of DBMS functionality. These interfaces define the boundaries of DBMS components such as record containers, query processors, and transaction coordinators that enable uniform, transactional access to data among such components. The proposed interfaces extend Microsoft's OLE/COM object services framework with database functionality, hence these interfaces are collectively referred to as OLE DB. The OLE DB functional areas include data access and updates (rowsets), query processing, schema information, notifications, transactions, security, and access to remote data. In a sense, OLE DB represents an effort to bring database technology to the masses. This paper presents an overview of the OLE DB approach and its areas of componentization.

Active OODBMSs must provide efficient support for event detection, composition and rule execution. Previous experience of building active capabilities on top of existing closed OODBMSs has proven to be ineffective. We propose instead an active OODBMS architecture where event detection and rule support are tightly integrated with the rest of the core OODBMS functionality. After presenting an analysis of the requirements of active OODBMSs, we discuss the event set, rule execution modes and lifespan of the events supported in our architecture. We also discuss event composition coupling relative to transaction boundaries. Since building an active OODBMS ex nihilo is extremely expensive, we are building the REACH (REal-time ACtive Heterogeneous) OODBMS by extending Texas Instruments' Open OODB toolkit. Open OODB is particularly well-suited for our purposes because it is the first DBMS whose architecture closely resembles the active database paradigm. It provides low-level event detection and invokes appropriate DBMS functionality as actions. We describe the architecture of the event detection and composition mechanisms, and the rule-firing process of the REACH active OODBMS, and show how these mechanisms interplay with the Open OODB core mechanisms.>

This paper lays the theoretical foundations for identifying classes of views and data partitioning strategies that allow efficient incremental view maintenance using “partial information” about the underlying base relations. We present necessary and sufficient conditions for determining how a materialized select-project-join view can be updated using only the view definition, the current view materialization, and the update (i.e., no base relations); and also how to update views using only the view definition, the update, and the updated base relation (i.e., not the other base relations). We generalize the above results to use an arbitrary subset of the base relations and the current view materialization. Our results are especially useful in distributed databases, in disconnected and mobile computing environments where the underlying database is not always accessible, and in data warehousing applications. Our techniques can be used to minimize remote data access and often to completely avoid remote access. This paper gives the intuition and theory necessary for identifying and designing views that are efficiently maintainable using partial information.

No abstract available.

This paper reports our experiences building the query optimizer for TI's Open OODB system. To the best of our knowledge, it is the first working object query optimizer to be based on a complete extensible optimization framework including logical algebra, execution algorithms, property enforcers, logical transformation rules, implementation rules, and selectivity and cost estimation. Our algebra incorporates a new materialize operator with its corresponding logical transformation and implementation rules that enable the optimization of path expressions. Initial experiments on queries obtained from the object query optimization literature demonstrate that our optimizer is able to derive plans that are as efficient as, and often substantially more efficient than, the plans generated by other query optimization strategies. These experiments demonstrate that our initial choices for populating each part of our optimization framework are reasonable. Our experience also shows that having a complete optimization framework is crucial for two reasons. First, it allows the optimizer to discover plans that cannot be revealed by exploring only the alternatives provided by the logical algebra and its transformations. Second, it helps and forces the database system designer to consider all parts of the framework and to maintain a good balance of choices when incorporating a new logical operator, execution algorithm, transformation rule, or implementation rule. The Open OODB query optimizer was constructed using the Volcano Optimizer Generator, demonstrating that this second-generation optimizer generator enables rapid development of efficient and effective query optimizers for non-standard data models and systems.

The Open OODB project, part of the DARPA Persistent Object Base (POB) Program, is an effort to build an open, extensible object-oriented database management system (OODB) in which database functionality can be tailored for particular applications within an incrementally improvable framework. The system is designed to serve both as a platform for research and as a testbed that can meet the needs of demanding, next generation database applications. The Open OODB project goals are to describe the design space of OODBs; build an architectural framework that enables configuring independently useful modules to form an OODB; verify the suitability of this open approach by implementing an OODB to these specifications; and identify opportunities for building consensus that can lead to much-needed OODB standards. The motivating factors in this approach were that our previous experience in object-oriented databases had convinced us that different applications have differing requirements, and that a monolithic system is unlikely to meet the needs of many demanding kinds of applications.

The problem of efficiently computing a joint between two base relations in the presence of queries and updates to the base relations is dealt with. A performance analysis of three methods: join index, materialized view, and hybrid-hash join, is presented. The first two methods are examples of a strategy based on data caching; they represent two ends of a spectrum of possibilities depending on the attributes projected in the materialization. The third method is an example of a conventional strategy for computing a join from base relations. The results of this study show that the method of choice depends on the environment, and in particular on the update activity on base relations, the join selectivity, and the amount of main memory available. A by-product of this study is a strategy for incrementally maintaining a join index in the presence of updates to the underlying base relations.>

Consider a database containing not only base relations but also stored derived relations (also called materialized or concrete views). When a base relation is updated, it may also be necessary to update some of the derived relations. This paper gives sufficient and necessary conditions for detecting when an update of a base relation cannot affect a derived relation (an irrelevant update), and for detecting when a derived relation can be correctly updated using no data other than the derived relation itself and the given update operation (an autonomously computable update). The class of derived relations considered is restricted to those defined by PSJ -expressions, that is, any relational algebra expressions constructed from an arbitrary number of project, select and join operations (but containing no self-joins). The class of update operations consists of insertions, deletions, and modifications, where the set of tuples to be deleted or modified is specified by a selection condition on attributes of the relation being updated.

Query processing can be sped up by keeping frequently accessed users' views materialized. However, the need to access base relations in response to queries can be avoided only if the materialized view is adequately maintained. We propose a method in which all database updates to base relations are first filtered to remove from consideration those that cannot possibly affect the view. The conditions given for the detection of updates of this type, called irrelevant updates, are necessary and sufficient and are independent of the database state. For the remaining database updates, a differential algorithm can be applied to re-evaluate the view expression. The algorithm proposed exploits the knowledge provided by both the view definition expression and the database update operations.