As applications and data evolve over time, the database schema must be adjusted to accommodate their needs. Schema changes in relational databases have traditionally required synchronization with concurrent read and write access, causing significant interruptions to user applications. Although, most commercial databases have optimized common schema changes to reduce their runtime, they have not fundamentally addressed the requirement for synchronization which can lead to data being inaccessible for minutes or even hours in the presence of long running queries. MD-MVCC is a new technology in Azure SQL Database that enables multi-version concurrency control for schema changes. This allows schema changes to occur without any synchronization with concurrent queries which can operate on the earlier version of the schema until ongoing operations are committed, following Snapshot Isolation semantics. Schema deployments can now occur with minimal impact, increasing data availability but also flexibility for application developers. This required a comprehensive redesign of the schema management and metadata components of the RDBMS that are now multi-versioned across all layers, from query execution and in-memory caches to the system tables where metadata is persisted. This paper presents the overall design of MD-MVCC and demonstrates how it fundamentally improves data availability during schema changes without incurring any performance overheads.

SQL Ledger is a new technology that allows cryptographically verifying the integrity of relational data stored in Azure SQL Database and SQL Server. This is achieved by maintaining all historical data in the database and persisting its cryptographic (SHA-256) digests in an immutable, tamper-evident ledger. Digests representing the overall state of the ledger can then be extracted and stored outside of the RDBMS to protect the data from any attacker or high privileged user, including DBAs, system and cloud administrators. The ledger and the historical data are managed transparently, offering protection without any application changes. Historical data is maintained in a relational form to support SQL queries for auditing, forensics and other purposes. SQL Ledger provides cryptographic data integrity guarantees while maintaining the power, flexibility and performance of a commercial RDBMS. In contrast to Blockchain solutions that aim for full integrity, SQL Ledger offers a form of integrity protection known as Forward Integrity. The proposed technology is significantly cheaper and more secure than traditional solutions that establish trust based on audits or mediators, but also has substantial advantages over Blockchain solutions that are complex to deploy, lack data management capabilities and suffer in terms of performance due to their decentralized nature.

This paper presents Always Encrypted, a recently released feature of Microsoft SQL Server that uses column granularity encryption to provide cryptographic data protection guarantees. Always Encrypted can be used to outsource database administration while keeping the data confidential from an administrator, including cloud operators. The first version of Always Encrypted was released in Azure SQL Database and as part of SQL Server 2016, and supported equality operations over deterministically encrypted columns. The second version, released as part of SQL Server 2019, uses an enclave running within a trusted execution environment to provide richer functionality that includes comparison and string pattern matching for an IND-CPA-secure (randomized) encryption scheme. We present the security, functionality, and design of Always Encrypted, and provide a performance evaluation using the TPC-C benchmark.

The database-as-a-service paradigm in the cloud (DBaaS) is becoming increasingly popular. Organizations adopt this paradigm because they expect higher security, higher availability, and lower and more flexible cost with high performance. It has become clear, however, that these expectations cannot be met in the cloud with the traditional, monolithic database architecture. This paper presents a novel DBaaS architecture, called Socrates. Socrates has been implemented in Microsoft SQL Server and is available in Azure as SQL DB Hyperscale. This paper describes the key ideas and features of Socrates, and it compares the performance of Socrates with the previous SQL DB offering in Azure.

Azure SQL Database and the upcoming release of SQL Server introduce a novel database recovery mechanism that combines traditional ARIES recovery with multi-version concurrency control to achieve database recovery in constant time, regardless of the size of user transactions. Additionally, our algorithm enables continuous transaction log truncation, even in the presence of long running transactions, thereby allowing large data modifications using only a small, constant amount of log space. These capabilities are particularly important for any Cloud database service given a) the constantly increasing database sizes, b) the frequent failures of commodity hardware, c) the strict availability requirements of modern, global applications and d) the fact that software upgrades and other maintenance tasks are managed by the Cloud platform, introducing unexpected failures for the users. This paper describes the design of our recovery algorithm and demonstrates how it allowed us to improve the availability of Azure SQL Database by guaranteeing consistent recovery times of under 3 minutes for 99.999% of recovery cases in production.

Azure SQL Database and the upcoming release of SQL Server enhance Online Index Rebuild to provide fault-tolerance and allow index rebuild operations to resume after a system failure or a user-initiated pause. SQL Server is the first commercial DBMS to support pause and resume functionality for index rebuilds. This is achieved by splitting the operation into incremental units of work and persisting the required state so that it can be resumed later with minimal loss of progress. At the same time, the proposed technology minimizes the log space required for the operation to succeed, making it possible to rebuild large indexes using only a small, constant amount of log space. These capabilities are critical to guarantee the reliability of these operations in an environment where a) the database sizes are increasing at a much faster pace compared to the available hardware, b) system failures are frequent in Cloud architectures using commodity hardware, c) software upgrades and other maintenance tasks are automatically handled by the Cloud platforms, introducing further unexpected failures for the users and d) most modern applications need to be available 24/7 and have very tight maintenance windows. This paper describes the design of "Resumable Online Index Rebuild" and discusses how this technology can be extended to cover more schema management operations in the future.