This article proposes an index key compression scheme based on the notion of distinction bits. It proves that the distinction bits of index keys are sufficient information to determine the sorted order of the index keys. The actual compression ratio may vary depending on the characteristics of datasets (an average of 2.76:1 compression ratio was observed in the authors’ experiments). However, the index key compression scheme leads to significant performance improvements during the reconstruction of large-scale indexes. This study’s index key compression can be effectively used for database replication and index recovery in modern main-memory database systems.

Modern in-memory database systems are facing the need of efficiently supporting mixed workloads of OLTP and OLAP. A conventional approach to this requirement is to rely on ETL-style, application-driven data replication between two very different OLTP and OLAP systems, sacrificing real-time reporting on operational data. An alternative approach is to run OLTP and OLAP workloads in a single machine, which eventually limits the maximum scalability of OLAP query performance. In order to tackle this challenging problem, we propose a novel database replication architecture called Asynchronous Parallel Table Replication (ATR). ATR supports OLTP workloads in one primary machine, while it supports heavy OLAP workloads in replicas. Here, row-store formats can be used for OLTP transactions at the primary, while column-store formats are used for OLAP analytical queries at the replicas. ATR is designed to support elastic scalability of OLAP query performance while it minimizes the overhead for transaction processing at the primary and minimizes CPU consumption for replayed transactions at the replicas. ATR employs a novel optimistic lock-free parallel log replay scheme which exploits characteristics of multi-version concurrency control (MVCC) in order to enable real-time reporting by minimizing the propagation delay between the primary and replicas. Through extensive experiments with a concrete implementation available in a commercial database system, we demonstrate that ATR achieves sub-second visibility delay even for update-intensive workloads, providing scalable OLAP performance without notable overhead to the primary.

As a great deal of data has been accumulated in various disciplines, the need for the integrative analysis of separate but relevant data sources is becoming more important. Combining data sources can provide global insight that is otherwise difficult to obtain from individual sources. Because of privacy, regulations, and other issues, many large-scale data repositories remain closed off from the outside, raising what has been termed the data silo issue. The huge volume of today's big data often leads to computational challenges, adding another layer of complexity to the solution. In this paper, we propose a novel method called collaborative analytics by ensemble learning (CABEL), which attempts to resolve the main hurdles regarding the silo issue: accuracy, privacy, and computational efficiency. CABEL represents the data stored in each silo as a compact aggregate of samples called the silo signature. The compact representation provides computational efficiency and privacy preservation but makes it challenging to produce accurate analytics. To resolve this challenge, we formulate the problem of attribute domain sampling and reconstruction, and propose a solution called the Chebyshev subset. To model collaborative efforts to analyze semantically linked but structurally disconnected databases, CABEL utilizes a new ensemble learning technique termed the weighted bagging of base classifiers. We demonstrate the effectiveness of CABEL by testing with a nationwide health-insurance data set containing approximately 4,182,000,000 records collected from the entire population of an Organisation for Economic Co-operation and Development (OECD) country in 2012. In our binary classification tests, CABEL achieved median recall, precision, and F-measure values of 89%, 64%, and 76%, respectively, although only 0.001–0.00001% of the original data was used for model construction, while maintaining data privacy and computational efficiency.

Business planning as well as analytics on top of large-scale database systems is valuable to decision makers, but planning operations known and implemented so far are very basic. In this paper we propose a new planning operation called interval disaggregate , which goes as follows. Suppose that the planner, typically the management of a company, plans sales revenues of its products in the current year. An interval of the expected revenue for each product in the current year is computed from historical data in the database as the prediction interval of linear regression on the data. A total target revenue for the current year is given by the planner. The goal of the interval disaggregate operation is to find an appropriate disaggregation of the target revenue, considering the intervals. We formulate the problem of interval disaggregation more precisely and give solutions for the problem. Multidimensional geometry plays a crucial role in the problem formulation and the solutions. We implemented interval disaggregation into the planning engine of SAP HANA and did experiments on real-world data. Our experiments show that interval disaggregation gives more appropriate solutions with respect to historical data than the known basic disaggregation called referential disaggregation. We also show that interval disaggregation can be combined with the deseasonalization technique when the dataset shows seasonal fluctuations.

The SAP HANA database is the core of SAP's new data management platform. The overall goal of the SAP HANA database is to provide a generic but powerful system for different query scenarios, both transactional and analytical, on the same data representation within a highly scalable execution environment. Within this paper, we highlight the main features that differentiate the SAP HANA database from classical relational database engines. Therefore, we outline the general architecture and design criteria of the SAP HANA in a first step. In a second step, we challenge the common belief that column store data structures are only superior in analytical workloads and not well suited for transactional workloads. We outline the concept of record life cycle management to use different storage formats for the different stages of a record. We not only discuss the general concept but also dive into some of the details of how to efficiently propagate records through their life cycle and moving database entries from write-optimized to read-optimized storage formats. In summary, the paper aims at illustrating how the SAP HANA database is able to efficiently work in analytical as well as transactional workload environments.

Detecting performance anomalies and finding their root causes are tedious tasks requiring much manual work. Functionality enhancements in DBMS development as in most software development often introduce performance problems in addition to bugs. To detect the problems as soon as they are introduced, which often happens during the early phases of a development cycle, we adopt performance regression testing early in the process. In this paper, we describe a framework that we developed to manage performance anomalies after establishing a set of conditions for a problem to be considered an anomaly. The framework uses Statistical Process Control (SPC) charts to detect performance anomalies and differential profiling to identify their root causes. By automating the tasks within the framework we were able to remove most of the manual overhead in detecting anomalies and reduce the analysis time for identifying the root causes by about 90 percent in most cases. The tools developed and deployed based on the framework allow us continuous, automated daily monitoring of performance in addition to the usual functionality monitoring in our DBMS development.

DIAS

Abstract With the proliferation of various geographic databases on the Internet, we have seen increasing needs for accessing them concurrently and remotely via the Web for high‐level decision making. In this paper, we present Middleware for Efficient Access to Databases through OpenGIS Wrappers (MEADOW), an object‐oriented middleware system we have developed to meet these needs. Current OpenGIS standard addresses many interoperability issues involved in such a global utilization of geographic databases. However, existing Simple Feature specification for CORBA (SFCORBA) implementations of OpenGIS proved to be insufficient for MEADOW. The main problems are the complexity of system development and maintenance, and the inefficiency of accessing remote data servers for processing region queries. We resolved the complexity problem by automatically generating a major portion of the application code, specifically wrappers on database servers and client library modules called transparent access providers. A MEADOW view definition language was developed as a high‐level specification language for this purpose. The efficiency problem was resolved by using a region‐based group prefetching of spatial objects from a geographic region. In addition, we implemented an OID‐based semijoin for efficient global query processing, and a region‐level locking to enhance the level of concurrency among region queries. Copyright © 2002 John Wiley & Sons, Ltd

With a GByte of memory priced at less than $2000, main-memory DBMSs (MMDBMSs) are emerging as an economically viable alternative to disk-resident DBMSs (DRDBMSs) in many problem domains. The MMDBMS can show significantly higher performance than the DRDBMS by reducing disk accesses to the sequential form of log writing and occasional checkpointing. Upon a system crash, the recovery process begins by accessing the disk-resident log and checkpoint data to restore a consistent state. With increasing CPU speed, however, such disk access is still the dominant bottleneck in MMDBMSs. To overcome this bottleneck, this paper explores alternatives of parallel logging and recovery. The major contribution of this paper is the so-called differential logging scheme that permits unrestricted parallelism in logging and recovery. Using the bit-wise XOR operation both to compute the differential log between the before and after images and to recover the consistent database state, this scheme offers the room for significant performance improvement in the MMDBMS. First, with logging done on the difference, the log volume is reduced to almost half compared with the conventional physical logging. Second, the commutativity and associativity of XOR enables processing of log records in an arbitrary order. This means that we can freely distribute log records to multiple disks to improve the logging performance. During the recovery time, we can do a parallel restart independently for each log disk. This paper shows the superior performance of the differential logging compared to the physical logging in a shared-memory multiprocessor environment.

Recent studies have shown that cache-conscious indexes such as the CSB+-tree outperform conventional main memory indexes such as the T-tree. The key idea of these cache-conscious indexes is to eliminate most of child pointers from a node to increase the fanout of the tree. When the node size is chosen in the order of the cache block size, this pointer elimination effectively reduces the tree height, and thus improves the cache behavior of the index. However, the pointer elimination cannot be directly applied to multidimensional index structures such as the R-tree, where the size of a key, typically, an MBR (minimum bounding rectangle), is much larger than that of a pointer. Simple elimination of four-byte pointers does not help much to pack more entries in a node.

With the proliferation of various geographic database servers on the Internet, the need to access them simultaneously through the Web arises frequently for high-level decision making. The ongoing OpenGIS standard addresses many of the interoperability issues to make such global utilization of geographic databases possible. Based on the OpenGIS standard, the paper presents an object oriented architecture for the efficient Web based access to multiple geographic databases on the Internet. Called MEADOW, it provides a pair of automatically generated modules: the OpenGIS wrapper on the server side and the matching transparent access provider (TAP) on the client side. In cooperation with the wrapper, TAP supports an application's efficient access to the databases through prefetching and caching of remote objects.

Throughout many research and development projects for active rule systems, active rules are implemented with different syntax and semantics. It becomes one of the stumbling blocks to apply active database systems especially in networked heterogeneous multidatabase environments. Utilizing the recent development of CORBA and ODMG standards, a middleware approach to provide active rule systems for heterogeneous ODBMS is presented in this paper. The active rule system described is applied for integrity maintenance of spatial objects. According to the events included in application programs, the active rules represented in ECA type are inserted into the program by a preprocessor. One advantage of this compile approach is that the preprocessed program can be compiled and executed without the overhead of runtime monitoring. For the changed rules after compilation, a run time interpreter is included in the executable program.

Article Free Access Share on Sibling clustering of tree-based spatial indexes for efficient spatial query processing Authors: Kihong Kim School of Electrical Eng., Seoul Nat'l Univ., #807 Kwanak P.O. Box 34, Seoul 151-742 korea School of Electrical Eng., Seoul Nat'l Univ., #807 Kwanak P.O. Box 34, Seoul 151-742 koreaView Profile , Sang K. Cha School of Electrical Eng., Seoul Nat'l Univ., #807 Kwanak P.O. Box 34, Seoul 151-742 korea School of Electrical Eng., Seoul Nat'l Univ., #807 Kwanak P.O. Box 34, Seoul 151-742 koreaView Profile Authors Info & Claims CIKM '98: Proceedings of the seventh international conference on Information and knowledge managementNovember 1998Pages 398–405https://doi.org/10.1145/288627.288686Published:01 November 1998Publication History 21citation368DownloadsMetricsTotal Citations21Total Downloads368Last 12 Months21Last 6 weeks4 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Xmas is an extensible main-memory storage system for high-performance embedded database applications. Xmas not only provides the core functionality of DBMS, such as data persistence, crash recovery, and concurrency control, but also pursues an extensible architecture to meet the requirements from various application areas. One crucial aspect of such extensibility is that an application developer can compose application-specific, high-level operations with a basic set of operations provided by the system. Called composite actions in Xmas, these operations are processed by a customized Xmas server with minimum interaction with application processes, thus improving the overall performance. This paper first presents the architecture and functionality of Xmas, and then demonstrates a simulation of mobile communication service.

This paper presents the architecture of Xmas, an extensible mainmemory storage system for high-performance and real-time database applications.Xmas not only provides the core functionality of a DBMS, such as data persistence, concurrency control, and crash recovery, but aIso has an extensible architecture to deal with requirements from various applications.One crucial aspect of such extensibility is that an application developer can compose applicationspecific, high-level operations with a basic set of operations provided by the system.CalIed composite actions in Xmas, these operations are processed by a customized Xmar server with minimum interaction with application processes, thus increasing the overall performance.

Kaleidoscope's approach is presented in the context of seeking improvement in the usability of interactive structured query language (SQL) interfaces. The system's cooperation is summarized as proposing valid query constituents step-by-step and providing lexical and semantic feedback immediately to users. To implement this intraquery guidance, the context-free grammar (CFG) is extended to capture the constraints useful for intraquery guidance, and the knowledge useful for pruning nonsensical queries and providing semantic feedback is articulated. For the SQL interface, this knowledge includes a strong domain concept, functional dependency, and integrity constraint rules, which can be acquired once in the database design step. The same types of knowledge are useful both for postquery cooperation and intraquery guidance. As SQL is supported bv virtually all database management system (DBMS) vendors, the approach presents a practical solution for casual database access.>

Querying databases to obtain information requires the user's knowledge of query language and underlying data. However, because the knowledge in human long-term memory is imprecise, incomplete, and often incorrect, user queries are subject to various types of failure. These may include spelling mistakes, the violation of the syntax and semantics of a query language, and the misconception of the entities and relationships in a database.