Many universities are introducing a new major in Data Science into their offering, to reflect the explosive growth in this field and the career opportunities it provides. As a field Data Science has elements from Computer Science and from Statistics, and curricula plans differ widely, both in the balance between the CS and Stats aspects, and also in the emphasis within the computing topics. This paper reports on the curriculum that has been taught for three years now at the University of Sydney. In particular, we describe the approach of a sequence of computing subjects which were developed specifically for the major, in order to bring students over several years to a sophisticated understanding of the data-handling aspects of Data Science. Students also take traditional subjects from both CS (such as Data Structures or AI) and from Statistics (such as Learning from Data and Statistical Inference). The data-centric specially-designed subjects we discuss in this paper are (i) Informatics: Data and Computation (in the first year), (ii) Big Data and Data Diversity (in the second year), and then upper-division subjects on (iii) Data Science Platforms, and (iv) Human-in-the-Loop Data Analytics.

MongoDB is a popular document store that is also available as a cloud-hosted service. MongoDB internally deploys primary-copy asynchronous replication, and it allows clients to vary the Read Preference, so reads can deliberately be directed to secondaries rather than the primary site. Doing this can sometimes improve performance, but the returned data might be stale, whereas the primary always returns the freshest data value. While state-of-practice is for programmers to decide where to direct the reads at application development time, they do not have full understanding then of workload or hardware capacity. It should be better to choose the appropriate Read Preference setting at runtime, as we describe in this paper. We show how a system can detect when the primary copy is saturated in MongoDB-as-a-Service, and use this to choose where reads should be done to improve overall performance. Our approach is aimed at a cloud-consumer; it assumes access to only the limited diagnostic data provided to clients of the hosted service.

Cloud vendors provide ready-to-use distributed DBMS solutions as a service. While the provisioning of a DBMS is usually fully automated, customers typically still have to make important design decisions which were traditionally made by the database administrator such as finding an optimal partitioning scheme for a given database schema and workload. In this paper, we introduce a new learned partitioning advisor based on Deep Reinforcement Learning (DRL) for OLAP-style workloads. The main idea is that a DRL agent learns the cost tradeoffs of different partitioning schemes and can thus automate the partitioning decision. In the evaluation, we show that our advisor is able to find non-trivial partitionings for a wide range of workloads and outperforms more classical approaches for automated partitioning design.

In this paper, we revisit the question of how ML algorithms can be best integrated into existing DBMSs to not only avoid expensive data copies to external ML tools but also to comply with regulatory reasons. The key observation is that database transactions already provide an execution model that allows DBMSs to efficiently mimic the execution model of modern parallel ML algorithms. As a main contribution, this paper presents DB4ML, an in-memory database kernel that allows applications to implement user-defined ML algorithms and efficiently run them inside a DBMS. Thereby, the ML algorithms are implemented using a programming model based on the idea of so called iterative transactions. Our experimental evaluation shows that DB4ML can support user-defined ML algorithms inside a DBMS with the efficiency of modern specialized ML engines. In contrast to DB4ML, these engines not only need to transfer data out of the DBMS but also hardcode the ML algorithms and thus are not extensible.

The SQL query language is the 'lingua franca' of transactional databases, and is essential for scalable data analytics. Learning SQL requires practical exercises with databases, including those parts of SQL with side-effects such as DDL statements, triggers, and stored procedures. Teaching this effectively to a non-technical audience is a challenge, especially in times of COVID-19 without face-to-face classes. The Grok Learning platform allows to design self-paced online tutorials with auto-graded exercises - but it was originally built for teaching programming languages. In this demo, we show how we extended the Grok platform to teach SQL for Data Scientists with comprehensive online learning. Grok supports a rich user interface with interactive examples where students can explore and experiment with each example query. This is ideal for learning declarative querying. Each query is executed in its own sandbox on a freshly initialised database instance which allows to teach all parts of SQL including DDL statements, stored procedures, triggers and UDFs. At the same time, the platform scales to thousands of concurrent users, while maintaining interactive response times.

ProgressiveDB transforms any standard SQL database into a progressive database capable of continuous, approximate query processing. It introduces a few small extensions to the SQL query language that allow clients to express progressive analytical queries. These extensions are processed in the ProgressiveDB middleware that sits between a database application and the underlying database providing interactive query processing as well as query steering capabilities to the user. In our demo, we show how this system allows a database application with a graphical user interface to interact with different backends, while providing the user with immediate feedback during exploratory data exploration of an on-time flight database. ProgressiveDB also supports efficient query steering by providing a new technique, called progressive views, which allows the intermediate results of one progressive query to be shared and reused by multiple concurrent progressive queries with refined scope.

Distributed data processing platforms for cloud computing are important tools for large-scale data analytics. Apache Hadoop MapReduce has become the de facto standard in this space, though its programming interface is relatively low-level, requiring many implementation steps even for simple analysis tasks. This has led to the development of more advanced dataflow oriented platforms, most prominently Apache Spark and Apache Flink. Those platforms not only aim to improve performance through improved in-memory processing, but in particular provide built-in high-level data processing functionality, such as filtering and join operators, which should make data analysis tasks easier to develop than with plain Hadoop MapReduce. But is this indeed the case? This paper compares three prominent distributed data processing platforms: Apache Hadoop MapReduce; Apache Spark; and Apache Flink, from a usability perspective. We report on the design, execution and results of a usability study with a cohort of master students, who were learning and working with all three platforms in order to solve different use cases set in a data science context. Our findings show that Spark and Flink are preferred platforms over MapReduce. Among participants, there was no significant difference in perceived preference or development time between both Spark and Flink as platforms for batch-oriented big data analysis. This study starts an exploration of the factors that make Big Data platforms more - or less - effective for users in data science.

Restful APIs are widely adopted in designing components that are combined to form web information systems. The use of REST is growing with the inclusion of smart devices and the Internet of Things, within the scope of web information systems, along with large-scale distributed NoSQL data stores and other web-based and cloud-hosted services. There is an important subclass of web information systems and distributed applications which would benefit from stronger transactional support, as typically found in traditional enterprise systems. In this paper, we propose REST+T (REST with Transactions), a transactional Restful data access protocol and API that extends HTTP to provide multi-item transactional access to data and state information across heterogeneous systems. We describe a case study called Tora, where we provide access through REST+T to an existing key-value store (WiredTiger) that was intended for embedded operation.

Typical cloud computing systems provide highly scalable and fault-tolerant data stores that may sacrifice other features like general multi-item transaction support. Recently techniques to implement multi-item transactions in these types of systems have focused on transactions across homogeneous data stores. Since applications access data in heterogeneous storage systems for legacy or interoperability reasons, we propose an approach that enables multi-item transactions with snapshot isolation across multiple heterogeneous data stores using only a minimal set of commonly implemented features such as single item consistency, conditional updates, and the ability to store additional meta-data. We define an client-coordinated transaction commitment protocol that does not rely on a central coordinating infrastructure. The application can take advantage of the scalability and fault-tolerance characteristics of modern key-value stores and access existing data in them, and also have multi-item transactional access guarantees with little performance impact. We have implemented our design in a Java library called Cherry Garcia (CG), that supports data store abstractions to Windows Azure Storage (WAS), Google Cloud Storage (GCS) and our own high-performance key-value store called Tora.

Since 1990's, Snapshot Isolation (SI) has been widely studied, and it was successfully deployed in commercial and open-source database engines. Berenson et al. showed that data consistency can be violated under SI. Recently, a new class of Serializable SI algorithms (SSI) has been proposed to achieve serializable execution while still allowing concurrency between reads and updates.

SUMMARY This paper describes a k‐mer approach to analysing DNA data and quickly answering certain types of ad hoc biological questions. These k‐mers (short DNA strings) are stored in a conventional relational database and indexed to support efficient exact match operations. We show that k‐mers around 20–25 bases long have interesting and useful uniqueness properties that can be used to compute a ‘relatedness’ metric and also allow k‐mers to be used as ‘unique enough’ tags to identify organisms and genes. This relatedness metric is used in SQL queries that can directly answer questions such as how two related species differ, and what genes are unique to an organism. The k‐mer tags have proven useful in applications, largely metagenomic ones that can quickly process large volumes of sequencing data to say something about what organisms and genes might be present in an environmental sample. All of this work is based on simple and fast exact matches of k‐mer strings using a database, rather than conventional alignment based on inexact matches of much longer strings. These k‐mer tools provide ways of rapidly exploring large genome spaces and handling large volumes of sequence data, and complement rather than replace existing alignment and assembly tools. Copyright © 2012 John Wiley & Sons, Ltd.

Snapshot Isolation (SI) is a popular concurrency control mechanism that has been implemented by many commercial and open-source platforms (e.g. Oracle, Postgre SQL, and MS SQL Server 2005). Unfortunately, SI can result in nonserializable execution, in which database integrity constraints can be violated. The literature reports some techniques to ensure that all executions are serializable when run in an engine that uses SI for concurrency control. These modify the application by introducing conflicting SQL statements. However, with each of these techniques the DBA has to make a choice among possible transactions to modify — and as we previously showed, making a bad choice of which transactions to modify can come with a hefty performance reduction. In this paper we propose a novel technique called ELM to introduce conflicts in a separate lock-manager object. Experiments with two platforms show that ELM has peak performance which is similar to SI, no matter which transactions are chosen for modification. That is, ELM is much less vulnerable from poor DBA choices than the previous techniques.

Many popular database management systems implement a multiversion concurrency control algorithm called snapshot isolation rather than providing full serializability based on locking. There are well-known anomalies permitted by snapshot isolation that can lead to violations of data consistency by interleaving transactions that would maintain consistency if run serially. Until now, the only way to prevent these anomalies was to modify the applications by introducing explicit locking or artificial update conflicts, following careful analysis of conflicts between all pairs of transactions. This article describes a modification to the concurrency control algorithm of a database management system that automatically detects and prevents snapshot isolation anomalies at runtime for arbitrary applications, thus providing serializable isolation. The new algorithm preserves the properties that make snapshot isolation attractive, including that readers do not block writers and vice versa. An implementation of the algorithm in a relational DBMS is described, along with a benchmark and performance study, showing that the throughput approaches that of snapshot isolation in most cases.

It is usually expected that performance is reduced by using stricter concurrency control, which reduces the risk of anomalies that can lead to data corruption. For example, the weak isolation level Read Committed allows anomalies that are prevented by two-phase locking (abbreviated 2PL), and because 2PL holds locks for longer than RC, it has lower throughput. We show that sometimes, guaranteed correctness can be obtained along with better throughput than RC, by use of the multiversion snapshot isolation mechanism along with modifications to application programs as proposed by Fekete et al. We investigate the conditions under which this effect occurs.

Several common DBMS engines use the multi- version concurrency control mechanism called Snapshot Isolation, even though application programs can experience non- serializable executions when run concurrently on such a platform. Several proposals exist for modifying the application programs, without changing their semantics, so that they are certain to execute serializably even on an engine that uses SI. We evaluate the performance impact of these proposals, and find that some have limited impact (only a few percent drop in throughput at a given multi-programming level) while others lead to much greater reduction in throughput of up-to 60% in high contention scenarios. We present experimental results for both an open- source and a commercial engine. We relate these to the theory, giving guidelines on which conflicts to introduce so as to ensure correctness with little impact on performance.

Many popular database management systems offer snapshot isolation rather than full serializability. There are well-known anomalies permitted by snapshot isolation that can lead to violations of data consistency by interleaving transactions that individually maintain consistency. Until now, the only way to prevent these anomalies was to modify the applications by introducing artificial locking or update conflicts, following careful analysis of conflicts between all pairs of transactions.

While the dominant approach to persistent storage in practice is to use a relational DBMS, there are some specialist applications that rely on object stores. The performance of these applications depends on the efficiency of the object store's concurrency control mechanism. Today's predominant concurrency control mechanism is strict two-phase object locking. In the 1980s, an interesting alternative was developed: commutativity-based semantic locking. In theory, it can outperform traditional locking schemes in certain scenarios with appropriate commutativity potential. In this paper, we study the real-world performance of different locking strategies in a particular industrial application from the telecommunications sector. We compare object-based locking and commutativity-based semantic locking. We found that, in this application, semantic locking performs equally to, but no better than, object locking in reasonable mixes of real- world transactions, and that it only outperforms in a deliberately contrived mix.

Data processing in wireless sensor networks often relies on high-speed data stream input, but at the same time is inherently constrained by limited resource availability. Thus, energy efficiency and good resource management are vital for in-network processing techniques. We propose enabling resource-awareness for in-network processing algorithms by means of a resource monitoring component and designed a corresponding framework. As proof of concept, we implement an online clustering algorithm, which uses the resource monitor to adapt to resource availability, on the Sun SPOT sensor nodes from Sun Microsystem. We refer to this adaptive clustering algorithm as extended resource-aware cluster (ERA-cluster). Finally, we report on the outcome of several experiments to evaluate the validity of our approach in terms of resource adaptiveness and accuracy of the ERA-cluster. Results show that ERA-cluster can effectively adapt to resource availability while maintaining acceptable level of accuracy.

We discuss the integration of on-line data stream clustering into a distributed query processor for wireless sensor networks (WSNs). Our approach is to combine an adaptive clustering algorithm with in-network data processing by introducing specialised query operators that implement stateful stream processing. We have implemented a testbed for an on-line clustering algorithm as part of a query processing system for the Sun SPOT sensor network platform. The paper discusses the design alternatives for continuous stream clustering in WSNs and for the integration of the resource-awareness to be able to trade result accuracy for resource consumption.

We investigate query routing techniques in a cluster of databases for a query-dominant environment. The objective is to decrease query response time. Each component of the cluster runs an off-the-shelf DBMS and holds a copy of the whole database. The cluster has a coordinator that routes each query to an appropriate component. Considering queries of realistic complexity, e.g., TPC-R, this article addresses the following questions: Can routing benefit from caching effects due to previous queries? Since our components are black-boxes, how can we approximate their cache content? How to route a query, given such cache approximations? To answer these questions, we have developed a cache-aware query router that is based on signature approximations of queries. We report on experimental evaluations with the TPC-R benchmark using our PowerDB database cluster prototype. Our main result is that our approach of cache approximation routing is better than state-of-the-art strategies by a factor of two with regard to mean response time.

When relational database systems were introduced twenty years ago (1980), they were an infrastructure and main platform for application development. With today's information systems, the database system is a storage manager, far away from the applications. Our vision is that hyperdatabases become available that move up and extend database concepts to a higher level, closer to the applications. A hyperdatabase manages distributed objects and software components as well as workflows, in analogy to a database system that manages data and transactions. In short, hyperdatabases, also called "higher order databases", will provide "higher order data independence", e.g., immunity of applications against changes in the implementation of components and workload transparency. They will be the infrastructure for distributed information systems engineering of the future, and they are an abstraction from the host of current infrastructures and middleware technology. The article elaborates on this vision and outlines concrete projects at ETHZ such as PowerDB, a database cluster project. It shows how an efficient document engine can be built on top of a database cluster. A further project studies transactional process management as a layer on top of database transactions. Image similarity and multimedia components is another project where a hyperdatabase coordinates specialized components such as feature extraction and indexing services in a distributed environment.

The CORBA standard, together with its service specifications, has gained considerable attention in recent years. The CORBA Object Query Service allows for declarative access to heterogeneous storage systems. We have come up with an implementation of this service called Harmony. The objective of the article is to provide a detailed description and quantitative assessment of Harmony. Its main technical characteristics are data-flow evaluation, bulk transfer and intra-query parallelism. To carry out the evaluation, we have classified data exchange between components of applications in several dimensions: one is to distinguish between point-, context- and bulk data access. We have compared Harmony with: (1) data access through application-specific CORBA objects, and (2) conventional client/server communication, i.e., Embedded SQL. Our results show that Harmony performs much better than Alternative 1 for bulk data access. Besides that, due to the features mentioned above, Harmony, performs approximately as well as conventional client/server communication mechanisms.