OLAP models can be categorized with two types: MOLAP (multidimensional OLAP) and ROLAP (relational OLAP). In particular, MOLAP is efficient in multidimensional computing at the cost of cube maintenance, while ROLAP reduces the data storage size at the cost of expensive multidimensional join operations. In this paper, we propose a novel Fusion OLAP model to fuse the multi-dimensional computing model and relational storage model together to make the best aspects of both MOLAP and ROLAP worlds. The Fusion OLAP model can be integrated into the state-of-the-art in-memory databases with additional surrogate key indexes and vector indexes. We compared the Fusion OLAP implementations with three leading analytical in-memory databases. Our comprehensive experimental results show that Fusion OLAP implementation can achieve up to 35%, 365% and 169% performance improvements based on the Hyper, Vectorwise and MonetDB databases respectively, for the Star Schema Benchmark (SSB) with scale factor 100.

OLAP models can be categorized with two types: MOLAP (multidimensional OLAP) and ROLAP (relational OLAP). In particular, MOLAP is efficient in multidimensional computing at the cost of cube maintenance, while ROLAP reduces the data storage size at the cost of expensive multidimensional join operations. In this paper, we propose a novel Fusion OLAP model to fuse the multidimensional computing model and relational storage model together to make the best aspects of both MOLAP and ROLAP worlds. This is achieved by mapping the relation tables into virtual multidimensional model and binding the multidimensional operations into a set of vector indexes to enable multidimensional computing on relation tables. The Fusion OLAP model can be integrated into the state-of-the-art in-memory databases with additional surrogate key indexes and vector indexes. We compared the Fusion OLAP implementations with three leading analytical in-memory databases. Our comprehensive experimental results show that Fusion OLAP implementation can achieve up to 35, 365, and 169 percent performance improvements based on the Hyper, Vectorwise, and MonetDB databases, respectively, for the Star Schema Benchmark (SSB) with scale factor 100.

With the advancement of sensor electronic devices, wireless sensor networks have attracted more and more attention. Range query has become a significant part of sensor networks due to its availability and convenience. However, It is challenging to process range query while still protecting sensitive data from disclosure. Existing work mainly focuses on privacy- preserving range query, but neglects the damage of collusion attacks, probability attacks and differential attacks. In this paper, we propose a privacy- preserving, energy-efficient and multi-dimensional range query protocol called PERQ, which not only achieves data privacy, but also considers collusion attacks, probability attacks and differential attacks. Generalized distance-based and modular arithmetic range query mechanism are used. In addition, a novel cyclic modular verification scheme is proposed to verify the data integrity. Extensive theoretical analysis and experimental results confirm the high performance of PERQ in terms of energy efficiency, security and accountability requirements.

High-frequency trading (HFT) has always been welcomed because it benefits not only personal benefits but also the whole social welfare. While the recent advance of portfolio selection in HFT market enables to bring about more profit, it yields much contended OLTP workloads. Featuring exploiting the abundant parallelism, transaction pipeline, the state-of-the-art concurrency control (CC) mechanism, however, suffers from limited concurrency confronted with HFT workloads. Its variants that enable more parallel execution by leveraging fine-grained contention information also take little effect. To solve this problem, we for the first time observe and formulate the source of restricted concurrency as harmful ordering of transaction statements. To resolve harmful ordering, we propose PARE, a pipeline-aware reordered execution, to improve application performance by rearranging statements in order of their degrees of contention. In concrete, two mechanisms are devised to ensure the correctness of statement rearrangement and identify the degrees of contention of statements, respectively. We also study the off-line reordering problem. We prove that this problem is NP-hard and present an off-line reordering approach to approximate the optimal reordering strategy. Experiment results show that PARE can improve transaction throughput and reduce transaction latency on HFT applications by up to an order of magnitude than the state-of-the-art CC mechanism.

Denormalization is a common tactic for enhancing performance of data warehouses. However, it is rarely used in main memory databases, which regards storage space as scarce resource. In this paper, we demonstrate that MMDB can actually benefit from the strategy of denormalization. We have created A-Store, a prototypical main-memory database system customized for star and snowflake schemas, which applies the strategy of denormalization to achieve highly efficient OLAP. Instead of resorting to fully materialized denormalization, A-Store applies a method called virtual denomalization, which allows query processing to be performed in a denormalized way, while without incurring additional space consumption.

Recommendation is an important issue in e-commerce systems. Conventional recommender algorithms, like the collaborative filtering recommendation algorithms, have been extensively studied and developed into a very mature stage, in which how to further promote user's favorite items and alleviate sparsity and cold start problem become increasingly important. In this paper, traditional recommender algorithms are promoted by exploring group's preference to mitigate the issues above and improve the predicting accuracy in both cases. Our work is based on the following observation. Users in the same group share common interests, given a group there are items that the group is most interested in, on the other hand, given some specific items there is the first-rate group which shows most preference compared to other groups. This leads to our proposed PromoRec algorithm which focuses on promoting items that users are most likely to prefer with sparse insensitivity since group enriches user's data largely. In a multi-dimensional space, we show how to efficiently compute (a) the most popular items for a target group and (b) the group which shows most interests in specific items. In addition, without the needs of available user group information, we propose an automatic classification algorithm based on users' similar interests. To improve the recommendation accuracy, we use additional item classification information to help determine the similarity between users. The experiment results confirm that our method significantly enhanced the traditional item recommendation algorithms especially while predicting ratings of promoted items for sparse users.

With the proliferation of graph-based applications, such as social network management and Web structure mining, update-intensive graph databases have become an important component of today’s data management platforms. Several techniques have been recently proposed to exploit locality on both data organization and computational model in graph databases. However, little investigation has been conducted on buffer management of graph databases. To the best of our knowledge, current buffer managers of graph databases suffer performance loss caused by unnecessary random I/O access. To solve this problem, we develop a novel batch replacement policy for buffer management. This policy enables us to maximally exploit sequential I/O to improve the performance of graph database. However, trivial solution produces impractical maintenance for replacement plan with maximal sequential I/O. To enable the policy, we first devise a segment tree-based buffer manager to efficiently maintain a optimal replacement plan. Unfortunately, segment tree-based solution becomes bottleneck in multi-core environment. To remedy this weakness, a B-tree-based buffer manager is further proposed. Extensive experiments on real-world and synthetic datasets demonstrate the superiority of our method.

Denormalization is a common tactic for enhancing performance of data warehouses, though its side-effect is quite obvious. Besides being confronted with update abnormality, denormalization has to consume additional storage space. As a result, this tactic is rarely used in main memory databases, which regards storage space, i.e., RAM, as scarce resource. Nevertheless, our research reveals that main memory database can benefit enormously from denormalization, as it is able to remarkably simplify the query processing plans and reduce the computation cost. In this paper, we present A-Store, a main memory OLAP engine customized for star/snowflake schemas. Instead of generating fully materialized denormalization, A-Store resorts to virtual denormalization by treating array indexes as primary keys. This design allows us to harvest the benefit of denormalization without sacrificing additional RAM space. A-Store uses a generic query processing model for all SPJGA queries. It applies a number of state-of-the-art optimization methods, such as vectorized scan and aggregation, to achieve superior performance. Our experiments show that A-Store outperforms the most prestigious MMDB systems significantly in star/snowflake schema based query processing.

The proliferation of location-based social networks, such as Foursquare and Facebook Places, offers a variety of ways to record human mobility, including user generated geo-tagged contents, check-in services, and mobile apps. Although trajectory data is of great value to many applications, it is challenging to analyze and mine trajectory data due to the complex characteristics reflected in human mobility, which is affected by multiple contextual information. In this paper, we propose a Multi-Context Trajectory Embedding Model, called MC-TEM, to explore contexts in a systematic way. MC-TEM is developed in the distributed representation learning framework, and it is flexible to characterize various kinds of useful contexts for different applications. To the best of our knowledge, it is the first time that the distributed representation learning methods apply to trajectory data. We formally incorporate multiple context information of trajectory data into the proposed model, including user-level, trajectory-level, location-level, and temporal contexts. All the context information is represented in the same embedding space. We apply MC-TEM to two challenging tasks, namely location recommendation and social link prediction. We conduct extensive experiments on three real-world datasets. Extensive experiment results have demonstrated the superiority of our MC-TEM model over several state-of-the-art methods.

With the pervasiveness of location-based social networks, it becomes increasingly important to consider the social characteristics of locations shared among persons. Several studies have been proposed to infer social strength by using trajectory similarity. However, these studies have two major shortcomings. First, they rely on the explicit co-occurrence of check-in locations. In this situation, a user pair of two friends who seldom share common locations or a user pair of two strangers who heavily share common visited locations will receive an unreliable estimation of the real social strength between them. Second, these studies do not consider how the overall trajectory patterns of users change with the varying of living styles. In this article, we propose a probabilistic generative model to mine latent lifestyle-related patterns from human trajectory data for inferring social strength. It can automatically learnfunctionality topicsconsisting of locations with similar service functions and transition probabilities over the set of functionality topics. Furthermore, a lifestyle is modeled as a unique transition probability matrix over the set of functionality topics. A user has a preference distribution over the set of lifestyles, and he or she is able to select over multiple lifestyles to adapt to different living contexts. The learned lifestyle-related patterns are subsequently used as features in a supervised learner for both strength estimation and link prediction. We conduct extensive experiments to evaluate the performance of the proposed method on two real-world datasets. The experimental results demonstrate the effectiveness of our proposed method.

It is insufficient to search temporal text by only focusing on either time attribute or keywords today as we pay close attention to the evolution of event with time. Both temporal and textual constraints need to be considered in one single query, called Top-k Interval Keyword Query (TIKQ).In this paper, we presents a cloud-based system named INK that supports efficient execution of TIKQs with appropriate effectiveness on Hadoop and HBase. In INK, an Adaptive Index Selector (AIS) is devised to choose the better execution plan for various TIKQs adaptively based on the proposed cost model, and leverage two novel hybrid index modules (TriI and IS-Tree) to combine keyword and interval filtration seamlessly.

This paper investigates the problem of efficiently computing responsibility for lineages of conjunctive queries with inequalities on databases. We classify the lineages of a class of queries with inequalities, called IQ queries, into path and composite lineages. We first compile path lineages into lineage graphs and transform lineage graphs into matrices. Then we reduce the problem of computing responsibility for path lineages to the shortest path problem, which can be solved by the dynamic programming algorithm in PTIME. We further prove composite lineages can be decomposed into path lineages for computing responsibility. Thus, our first main result shows it is in PTIME to compute responsibility for lineages of IQ queries. We generalize the previous results on dichotomy of responsibility analysis for lineages of conjunctive queries with equalities, now in the presence of inequalities. After decomposing composite lineages into path lineages, the data population needed for computing responsibility decreases more than one order of magnitude. Thus, our algorithm can efficiently compute responsibility for composite lineages. In order to compute responsibility for lineages in general, we introduce a greedy algorithm, consisting of a reduction to the set cover problem. Finally, we demonstrate the benefits of the proposed algorithms with extensive experimental results.

In this work, we study a novel query type, called top-k,m queries. Suppose we are given a set of groups and each group contains a set of attributes, each of which is associated with a ranked list of tuples, with ID and score. All lists are ranked in decreasing order of the scores of tuples. We are interested in finding the best combinations of attributes, each combination involving one attribute from each group. More specifically, we want the top-k combinations of attributes according to the corresponding top-m tuples with matching IDs. This problem has a wide range of applications from databases to search engines on traditional and non-traditional types of data (relational data, XML, text, etc.). We show that a straightforward extension of an optimal top-k algorithm, the Threshold Algorithm (TA), has shortcomings in solving the km problem, as it needs to compute a large number of intermediate results for each combination and reads moreinputs than needed. To overcome this weakness, we provide here, for the first time, a provably instance-optimal algorithm and further develop optimizations for efficient query evaluation to reduce computational and memory costs and the number of accesses. We demonstrate experimentally the scalability and efficiency of our algorithms over three real applications.

SUMMARY With the rapid developments in video processing technologies, video data have increased rapidly and become popular in our daily life for both professional and consumer applications such as surveillance, education and entertainment. Because of the increasing processing workload, more and more queries with expensive video predicates are being implemented in a parallel environment for better performance. Such requirements entail that the data management system not only be able to store and access video content, but also be able to optimize queries that have expensive video predicates in an effective and efficient way in a cloud environment. In previous research literatures, parallel and distributed policies and query optimizations in relational database management systems are often based on the disk input/output (I/O) cost of involved operations and network transmission cost. However, for a query that contains expensive video predicates in a cloud environment, the traditional cost estimation model does not work well. Although researchers have proposed some approaches that can solve the problem in certain situations, there are still some unresolved issues, and these approaches need further optimizations. This paper is motivated by a real‐world large supermarket business data and video surveillance data management scenario in a parallel environment. By considering the characteristics of video data and their expensive processing, we present methods named operating results buffer and operating results buffer‐C for implementing expensive video predicates at simple node, mapping video data and executing expensive video predicates in a cloud environment, which reduce the cost of video data transmission and the invoking times of expensive video predicates. We propose a novel query optimization approach that reconstructs the join order‐based estimation for attribute cardinality and computes the total cost with I/O, network and expensive processing. This approach reduces the invoking times of expensive video predicates to a greater degree and gives a better solution for mixed query optimization, which contains traditional data types and large object operations in a cloud environment. Our query performance improves by 30% to 80% compared with existing expensive predicates query optimization methods. Copyright © 2011 John Wiley & Sons, Ltd.

Big data analysis is a main challenge we meet recently. Cloud computing is attracting more and more big data analysis applications, due to its well scalability and fault-tolerance. Some aggregation functions, like SUM, can be computed in parallel, because they satisfy distributive law of addition. Unfortunately, some of statistical functions are not naturally parallelizable. That means they do not satisfy distributive law of addition. In this paper, we focus on percentile computing problem. We proposed an iterative-style prediction-based parallel algorithm in a distributed system. Prediction is done through a sampling technique. Experiment results verify the efficiency of our algorithm.

In this paper, we analyse the data access characteristics of a typical XML information retrieval system and propose a new query aware buffer replacement algorithm based on prediction of Minimum Reuse Distance (MRD for short). The algorithm predicts an object's next reference distance according to the retrieval system's running status and replaces the objects that have maximum reuse distances. The factors considered in the replacement algorithm include the access frequency, creation cost, and size of objects, as well as the queries being executed. By taking into account the queries currently running or queuing in the system, MRD algorithm can predict more accurately the reuse distances of index data objects.

Video has become popular in our daily life for both professional and consumer applications. Both low level video processing and high level semantic video analysis are critically computational tasks in application domains. Most of current video computing tools are developed for specific analytic tasks, they are lack higher level interoperability with database and treat database merely as a relational data storage engine rather than an analytic platform, which causes inefficient data access and massive amount of data movement. In this paper, we study how to support video data management over relational database, and present our initial solutions of video data storage mechanism, video data access method and efficient video analytics. We also illustrate our ongoing prototype system HybVideo that developed in a novel architecture. It integrates above solutions to tackle the major challenges of providing a platform for both storage and analysis of video data.

This paper presents a reference framework, called BUD, to manage a large shared bank of unstructured data. This paper lists several important issues on managing or maintaining the unstructured data in BUD. BUD stores and manages the ever-growing unstructured data by introducing a novel technique called free-table, which is a conceptual view for end-users and a physical entity maintained by transactional storage manager of BUD. Free-table is cell-oriented but not column-oriented as relational table. It can store various types of unstructured data in cell with different versions. Additionally, we study two cases, VMP and PXRDB, to show that our proposal is feasible and tractable.

The requirements of OLAP applications increase rapidly by dramatically increased data volume, users, query volume and query complexity. The requirement for shortening update period in data warehouse is another crucial factor for a scalable OLAP application. In this paper, we propose a scalable OLAP prototype to support the query processing with increasing data volume by distributing the whole fact tuples to multiple servers to construct a set of sibling cubes which can be merged together to obtain the whole cube. We employ a light weight distribution policy with fully duplicated dimension tables in each sibling server on the observation of very low proportion of space cost for dimension tables. OLAP query with distributed aggregate functions can be transformed into queries to be performed parallel in sibling servers. For non-distributed computing aggregate functions, such as median, the optimized median aggregate computing algorithm is proposed to reduce transmission volume between servers while computing the global median values. We also present a three-level framework in data warehouse to meet the requirement of shorter update period in "operational business intelligence". An asynchronous tunnel model is proposed to reduce update latency by pre-fetching updated tuples to OLAP processing server. Finally, we set up prototype system ParaCube to evaluate performance in SN (shared-nothing) system and multi-core platforms.

In update intensive applications, main memory database systems produce large volume of log records, it is critical to write out the log records efficiently to speedup transaction processing. We propose a parallel recovery scheme based on XOR differential logging for main memory database systems in such environments. Some NVRAM is used to temporarily hold log records and decouple transaction committing from disk writes, inherited parallelism properties of differential logging are exploited to accelerate log flushing by using multiple log disks. During recovery, log records are loaded from multiple log disks and applied to data partition in time without the need of reordering according to serialization order, total recovery time is cut down. The scheme employs a data partition based consistent checkpointing method. The log records are classified according to IDs of data partitions accessed. Data partitions are recovered according to loading priorities computed from update frequencies and transaction waiting times, data access demands of new transactions coming after failure recovery are given attention immediately, thus the scheme provides system availability during recovery, which is of importance for large scale main memory database systems.

Detecting and exploiting correlations among columns in relational databases are of great value for query optimizers to generate better query execution plans (QEPs). We propose a more robust and informative metric, namely, entropy correlation coefficients, other than chi-square test to detect correlations among columns in large datasets. We introduce a novel yet simple kind of multi-dimensional synopses named COCA-Hist to cope with different correlations in databases. With the aid of the precise metric of entropy correlation coefficients, correlations of various degrees can be detected effectively; when correlation coefficients testify to mutual independence among columns, the AVI (attribute value independence) assumption can be adopted undoubtedly. COCA can also serve as a data-mining tool with superior qualities as CORDS does. We demonstrate the effectiveness and accuracy of our approach by several experiments.

It is widely realized that the integration of database and information retrieval techniques will provide users with a wide range of high quality services. In this paper, we study processing an l-keyword query, p1, p2, ···, pl, against a relational database which can be modeled as a weighted graph, G(V, E). Here V is a set of nodes (tuples) and E is a set of edges representing foreign key references between tuples. Let Vi V be a set of nodes that contain the keyword pi. We study finding top-k minimum cost connected trees that contain at least one node in every subset Vi, and denote our problem as GST-k. When k = 1, it is known as a minimum cost group Steiner tree problem which is NP-Complete. We observe that the number of keywords, l, is small, and propose a novel parameterized solution, with l as a parameter, to find the optimal GST-1, in time complexity O(3ln + 2l((l + log n)n + m)), where n and m are the numbers of nodes and edges in graph G. Our solution can handle graphs with a large number of nodes. Our GST-1 solution can be easily extended to support GST-k, which outperforms the existing GST-k solutions over both weighted undirected/directed graphs. We conducted extensive experimental studies, and report our finding.

The concept of dominance has recently attracted much interest in the context of skyline computation. Given an N-dimensional data set S, a point p is said to dominate q if p is better than q in at least one dimension and equal to or better than it in the remaining dimensions. In this paper, we propose extending the concept of dominance for business analysis from a microeconomic perspective. More specifically, we propose a new form of analysis, called Dominant Relationship Analysis (DRA), which aims to provide insight into the dominant relationships between products and potential buyers. By analyzing such relationships, companies can position their products more effectively while remaining profitable.To support DRA, we propose a novel data cube called DADA (Data Cube for Dominant Relationship Analysis), which captures the dominant relationships between products and customers. Three types of queries called Dominant Relationship Queries (DRQs) are consequently proposed for analysis purposes: 1)Linear Optimization Queries (LOQ), 2)Subspace Analysis Queries (SAQ), and 3)Comparative Dominant Queries (CDQ). Algorithms are designed for efficient computation of DADA and answering the DRQs using DADA. Results of our comprehensive experiments show the effectiveness and efficiency of DADA and its associated query processing strategies.

Data cube pre-computation is an important concept for supporting OLAP(Online Analytical Processing) and has been studied extensively. It is often not feasible to compute a complete data cube due to the huge storage requirement. Recently proposed quotient cube addressed this issue through a partitioning method that groups cube cells into equivalence partitions. Such an approach is not only useful for distributive aggregate functions such as SUM but can also be applied to the holistic aggregate functions like MEDIAN.Maintaining a data cube for holistic aggregation is a hard problem since its difficulty lies in the fact that history tuple values must be kept in order to compute the new aggregate when tuples are inserted or deleted. The quotient cube makes the problem harder since we also need to maintain the equivalence classes. In this paper, we introduce two techniques called addset data structure and sliding window to deal with this problem. We develop efficient algorithms for maintaining a quotient cube with holistic aggregation functions that takes up reasonably small storage space. Performance study shows that our algorithms are effective, efficient and scalable over large databases.

Replication is one of the key technologies in promoting the performance of mobile database systems. In this paper, a novel mobile database replication scheme, the transaction-level result-set propagation (TLRSP) model, is put forward. A conflict detection and resolution strategy based on TLRSP is discussed in detail and its implementation algorithm is proposed. In the TLRSP model, mobile users are allowed to access local replicas of the database and to submit local transactions when the system is disconnected. The locally committed transactions are sent to a fixed database server for conflict reconciliation and result-set incorporation when the system is reconnected. The TLRSP model uses the incremental refreshing method to synchronize database replicas and to maintain the consistency of the replicated mobile database system.