Most of mobile object trajectory clustering analysis to date has been focused on clustering the location points or sub-trajectories extracted from trajectory data. This paper presents TraceMob, a systematic approach to clustering whole trajectories of mobile objects traveling in road networks. TraceMob as a whole trajectory clustering framework has three unique features. First, we design a quality measure for the distance between two whole trajectories. By quality, we mean that the distance measure can capture the complex characteristics of trajectories as a whole including their varying lengths and their constrained movement in the road network space. Second, we develop an algorithm that transforms whole trajectories in a road network space into multidimensional data points in a euclidean space while preserving their relative distances in the transformed metric space. This transformation enables us to effectively shift the clustering task for whole mobile object trajectories in the complex road network space to the traditional clustering task for multidimensional data in a euclidean space. Third, we develop a cluster validation method for evaluating the clustering quality in both the transformed metric space and the road network space. Extensive experimental evaluation with trajectories generated on real road network maps of different cities shows that TraceMob produces higher quality clustering results and outperforms existing approaches by an order of magnitude.

Mining trajectory data has been gaining significant interest in recent years. However, existing approaches to trajectory clustering are mainly based on density and Euclidean distance measures. We argue that when the utility of spatial clustering of mobile object trajectories is targeted at road-network aware location-based applications, density and Euclidean distance are no longer the effective measures. This is because traffic flows in a road network and the flow-based density characterization become important factors for finding interesting trajectory clusters. We propose NEAT-a road-network aware approach for fast and effective clustering of trajectories of mobile objects traveling in road networks. Our approach carefully considers the traffic locality characterized by the physical constraints of the road network, the traffic flow among consecutive road segments, and the flow-based density to organize trajectories into spatial clusters in a comprehensive three-phase clustering framework. NEAT discovers spatial clusters as groups of sub-trajectories which describe both dense and highly continuous flows of mobile objects. We perform extensive experiments with mobility traces generated using different scales of real road networks. Experimental results demonstrate the flexibility of the NEAT system and show that NEAT is highly accurate and runs orders of magnitude faster than existing density-based trajectory clustering approaches.

Mining trajectory data has been gaining significant interest in recent years. However, existing approaches to trajectory clustering are mainly based on density and Euclidean distance measures. We argue that when the utility of spatial clustering of mobile object trajectories is targeted at road network aware location based applications, density and Euclidean distance are no longer the effective measures. This is because traffic flows in a road network and the flow-based density characterization become important factors for finding interesting trajectory clusters of mobile objects travelling in road networks. In this paper, we propose NEAT-a road network aware approach for fast and effective clustering of spatial trajectories of mobile objects travelling in road networks. Our method takes into account the physical constraints of the road network, the network proximity and the traffic flows among consecutive road segments to organize trajectories into spatial clusters. The clusters discovered by NEAT are groups of sub-trajectories which describe both dense and highly continuous traffic flows of mobile objects. We perform extensive experiments with mobility traces generated using different scales of real road network maps. Our experimental results demonstrate that the NEAT approach is highly accurate and runs orders of magnitude faster than existing density-based trajectory clustering approaches.

Parallel processing is the key to speedup performance and to achieve high throughput in processing large scale data analytical workloads. However, failures of nodes involved in the analytical query can interrupt the whole process, resulting in the complete restart of the query if the system does not

Hive is a data warehousing solution on top of the Hadoop MapReduce framework that has been designed to handle large amounts of data and store them in tables like a relational database management system or a conventional data warehouse while using the parallelization and batch processing functionalities of the Hadoop MapReduce framework to speed up the execution of queries. Data inserted into Hive is stored in the Hadoop FileSystem (HDFS), which is part of the Hadoop MapReduce framework. To make the data accessible to the user, Hive uses a query language similar to SQL, which is called HiveQL. When a query is issued in HiveQL, it is translated by a parser into a query execution plan that is optimized and then turned into a series of map and reduce iterations. These iterations are then executed on the data stored in the HDFS, writing the output to a file.

Stream data analysis differs significantly from traditional data processing. To process the data online the algorithm has to work in one pass, incorporating new data into a model maintained in main memory. Storing a model or synopsis of processed data in the memory, which we call "data compression", is an important technique in both incremental and differential stream mining. This paper proposes several data compression schemes in one-pass categorical data clustering, and demonstrates their performance on synthetic and real data. Our compression schemes can efficiently generate compact representations of original data, so as to enable the algorithm to process streams at high speed and detect the changes in underlying data. The example algorithm based on these compression schemes achieves good accuracy in short execution time.

K-means is one of the most popular clustering algorithms. We introduce an efficient disk-based implementation of K-means. The proposed algorithm is designed to work inside a relational database management system. It can cluster large data sets having very high dimensionality. In general, it only requires three scans over the data set. It is optimized to perform heavy disk I/O and its memory requirements are low. Its parameters are easy to set. An extensive experimental section evaluates quality of results and performance. The proposed algorithm is compared against the Standard K-means algorithm as well as the Scalable K-means algorithm.

The visual exploration of large databases calls for a tight coupling of database and visualization systems. Current visualization systems typically fetch all the data and organize it in a scene tree, which is then used to render the visible data. For immersive data explorations, where an observer navigates in a potentially huge data space and explores selected data regions this approach is inadequate. A scalable approach is to make the database system observer-aware and exchange the data that is visible and most relevant to the observer.In this paper we present iTopN an incremental algorithm for extracting the most visible objects relative to the current position of the observer. We implement iTopN and compare it to an improved version of the R-tree that extends LRU with the caching of the top levels of the R-tree (LW-LRU). Our experiments show that iTopN is orders of magnitude faster than LW-LRU given the same amount of memory. Our experiments also show that for LW-LRU to perform as fast as iTopN it needs three times as much memory.

Data mining is defined as the process of discovering significant and potentially useful patterns in large volumes of data. Discovering associations between items in a large database is one such data mining activity. In finding associations, support is used as an indicator as to whether an association is interesting. In this paper, we discuss three alternative interest measures for associations: any-confidence, all-confidence, and bond. We prove that the important downward closure property applies to both all-confidence and bond. We show that downward closure does not hold for any-confidence. We also prove that, if associations have a minimum all-confidence or minimum bond, then those associations will have a given lower bound on their minimum support and the rules produced from those associations will have a given lower bound on their minimum confidence as well. However, associations that have that minimum support (and likewise their rules that have minimum confidence) may not satisfy the minimum all-confidence or minimum bond constraint. We describe the algorithms that efficiently find all associations with a minimum all-confidence or minimum bond and present some experimental results.

Clustering is a fundamental Data Mining technique. This article presents an improved EM algorithm to cluster large data sets having high dimensionality, noise and zero variance problems. The algorithm incorporates improvements to increase the quality of solutions and speed. In general the algorithm can find a good clustering solution in 3 scans over the data set. Alternatively, it can be run until it converges. The algorithm has a few parameters that are easy to set and have defaults for most cases. The proposed algorithm is compared against the standard EM algorithm and the On-Line EM algorithm.

We consider the use of data reduction techniques for the problem of approximate query answering. We focus on applications for which accurate answers to selective queries are required, and for which the data are very high dimensional (having hundreds of attributes). We present a new data reduction method for this type of application, called the RS kernel. We demonstrate the effectiveness of this method for answering difficult, highly selective queries over high dimensional data using several real datasets.

We introduce the linear file template, which is a generic data organization suitable for use with many different types of data. The linear file is specifically designed to handle intense database update loads concurrently with processing of analytic queries.

Broadcast is a scalable way of disseminating data because broadcasting an item satisfies all outstanding client requests for it. However, because the transmission medium is shared, individual requests may have high response times. In this paper, we show how to minimize the average response time given multiple broadcast channels by optimally partitioning data among them. We also offer an approximation algorithm that is less complex than the optimal and show that its performance is near-optimal for a wide range of parameters. Finally, we briefly discuss the extensibility of our work with two simple, yet seldom researched extensions, namely, handling varying sized items and generating single channel schedules.

To avoid the high cost of continuous connectivity, a class of mobile applications employs replicas of shared data that are periodically updated. Updates to these replicas are typically performed on a client-by-client basis--that is, the server individually computes and transmits updates to each client--limiting scalability. By basing updates on replica groups (instead of clients), however, update generation complexity is no longer bound by client population size. Clients then download updates of pertinent groups. Proper group design reduces redundancies in server processing, disk usage and bandwidth usage, and dimininishes the tie between the complexity of updating replicas and the size of the client population. In this paper, we expand on previous work done on group design, include a detailed I/O cost model for update generation, and propose a heuristic-based greedy algorithm for group computation. Experimental results with an adapted commercial replication system demonstrate a significant increase in overall scalability over the client-centric approach.

This work describes our experiences in discovering association rules in medical data to predict heart disease. We focus on two aspects of this work: mapping medical data to a transaction format suitable for mining association rules, and identifying useful constraints. Based on these aspects we introduce an improved algorithm to discover constrained association rules. We present an experimental section explaining several interesting discovered rules.

Clustering is a data mining problem that has received significant attention by the database community. Data set size, dimensionality and sparsity have been identified as aspects that make clustering more difficult. The article introduces a fast algorithm to cluster large binary data sets where data points have high dimensionality and most of their coordinates are zero. This is the case with basket data transactions containing items, that can be represented as sparse binary vectors with very high dimensionality. An experimental section shows performance, advantages and limitations of the proposed approach.

Mining for association rules is considered an important data mining problem. Many different variations of this problem have been described in the literature. We introduce the problem of mining for negative associations. A naive approach to finding negative associations leads to a very large number of rules with low interest measures. We address this problem by combining previously discovered positive associations with domain knowledge to constrain the search space such that fewer but more interesting negative rules are mined. We describe an algorithm that efficiently finds all such negative associations and present the experimental results.

Client-server systems cache data in client buffers to deliver good performance. Several efficient protocols have been proposed to maintain the coherence of the cached data. However, none of the protocols distinguish between index pages and data pages. We propose a new coherence protocol, called relaxed index consistency, that exploits the inherent differences in the coherence and concurrency control (C&CC) requirements for index and data pages. The key idea is to incur a small increase in computation time at the clients to gain a significant reduction in the number of messages exchanged between the clients and the servers. The protocol uses concurrency control on data pages to maintain the coherence of index pages. A performance-conscious implementation of the protocol that makes judicious use of version numbers is proposed. We show, through both qualitative and quantitative analyses, the performance benefits of making the distinction between index pages and data pages for the purposes of C&CC. Our simulation studies show that the relaxed index consistency protocol improves system throughput by as much as 15% to 88%, based on the workload.

Whenever data is moved across nodes in the parallel database system, the indexes need to be modified too. Index modification overhead can be quite severe because there can be a large number of indexes on a relation. In this paper, we study two alternatives to index modification, namely OAT (One-At-a-Time page movement) and BULK (bulk page movement). OAT and BULK are two extremes on the spectrum of the granularity of data movement. OAT and BULK differ in two respects: first, OAT uses very little additional disk space (at most one extra page), whereas BULK uses a large amount of disk space. Second, BULK uses sequential prefetch I/O to optimize on the number of I/Os during index modification, while OAT does not. Using an experimental testbed, we show that BULK is an order of magnitude faster than OAT. In terms of the impact on transaction performance during reorganization, BULK and OAT perform differently: when the number of indexes to be modified is either one or two, OAT has a lesser impact on the transaction performance degradation. However, when the number of indexes is greater than two, both techniques have the same impact on transaction performance.

Many new applications are being developed that capitalize on recent advances in multimedia systems. Such systems provide users with access to images, video clips, and audio samples as well as traditional text. A major challenge is to store the data for continuous media such as video and audio in such a way as to guarantee delay-free user input and output. We propose a hybrid solution to this challenge which combines the use of I/O parallelism with merging techniques for continuous media. These parallelism with merging (PWM) techniques outperformed techniques based solely on parallelism in simulation experiments.

Multiple-disk I/O systems (disk arrays) have been an attractive approach to meet high performance I/O demands in data intensive applications such as information retrieval systems. When we partition and distribute files across multiple disks to exploit the potential for I/O parallelism, a balanced I/O workload distribution becomes important for good performance. Naturally, the performance of a parallel information retrieval system using an inverted file structure is affected by the partitioning scheme of the inverted file. In this paper, we propose two different partitioning schemes for an inverted file system for a shared-everything multiprocessor machine with multiple disks. We study the performance of these schemes by simulation under a number of workloads where the term frequencies in the documents are varied, the term frequencies in the queries are varied, the number of disks are varied and the multiprogramming level is varied.>

Optimizing large join queries that consist of many joins has been recognized as NP-hard. Most of the previous work focuses on a uniprocessor environment. In a multiprocessor, the location of each join adds another dimension to the complexity of the problem. In this paper, we examine the feasibility of exploiting the inherent parallelism in optimizing large join queries on a hypercube multiprocessor. This includes using the multiprocessor not only to answer the large join query but also to optimize it. We propose an algorithm to estimate the cost of a parallel large join plan. Three heuristics are provided for generating an initial solution, which is further optimized by an iterative local-improvement method. The entire process of parallel query optimization and execution is simulated on an Intel iPSC/2 hypercube machine. Our experimental results show that the performance of each heuristic depends on the characteristics of the query.>

Presents a simulation-based performance analysis of a concurrent file reorganization algorithm. We examine the effect on throughput of (a) buffer size, (b) degree of reorganization, (c) write probability of transactions, (d) multiprogramming level, and (e) degree of clustered transactions. The problem of file reorganization that we consider involves altering the placement of records on pages of a secondary storage device. In addition, we want this reorganization to be done in place, i.e. using the file's original storage space for the newly reorganized file. Our approach is appropriate for a non-in-place reorganization as well. The motivation for such a physical change, i.e. record clustering, is to improve the database system's performance, i.e. minimizing the number of page accesses made in answering a set of queries. There are numerous record clustering algorithms, but they usually do not solve the entire problem, i.e., they do not specify how to efficiently reorganize the file to reflect the clustering assignment that they determine. In previous work, we have presented an algorithm that is a companion to general record clustering algorithms, i.e. it actually transforms the file. In this work we show through simulation that our algorithm, when run concurrently with user transactions, provides an acceptable level of overall database system performance.>

An indexing scheme called partitioned global indexes (PGI) for a locally distributed database system is presented. The scheme builds a global index for the entire relation and partitions the index across the sites. A strategy for processing such an index is also presented. In order to evaluate the performance of the scheme, a simulation model is developed. The simulation results are compared to the classical scheme, called partial indexes (PI), in which corresponding index and data entries are stored at the same site. The advantages and disadvantages of the indexing schemes when processing conjuctive queries are analytically investigated. Analysis and simulation experiments show that tradeoffs between the new and the classical scheme.>

The authors presents performance analysis of a concurrent file reorganization algorithm. They examined the effect of buffer size, degree of reorganization, and write probability of transactions on system throughput. The problem of file reorganization considered involves altering the placement of records on pages on a secondary storage device. This reorganization must be done in-place. The approach is appropriate for a non-in-place reorganization. The motivation for such a physical change is to improve the database system's performance, by minimizing the number of page accesses made in answering a set of queries. It is shown through simulation that the algorithm, when run concurrently with user transactions, provides an acceptable level of overall database system performance. >

The cube adaptive-hash join algorithm, which combines the merits of nested-loop and hybrid-hash, is presented. The performance of these algorithms is compared through analytical cost modeling. The nonuniform data value distribution of the inner relation is shown to have a greater impact than that of the outer relation. The cube adaptive-hash algorithm outperforms the cube hybrid-hash algorithm when bucket overflow occurs. In the worst case, this algorithm converges to the cube nested-loop-hash algorithm. When there is no hash table overflow, the cube adaptive-hash algorithm converges to the cube hybrid-hash algorithm. Since the cube adaptive-hash algorithm adapts itself depending on the characteristics of the relations, it is relatively immune to the data distribution.>

We present an efficient heuristic algorithm for record clustering that can run on a SIMD machine. We introduce the P-tree, and its associated numbering scheme, which in the split phase allows each processor independently to compute the unique cluster number of a record satisfying an arbitrary query. We show that by restricting ourselves in the merge phase to combining only sibling clusters, we obtain a parallel algorithm whose speedup ratio is optimal in the number of processors used. Finally, we report on experiments showing that our method produces substantial savings in an enviornment with relatively little overlap among the queries.

The problem of database normalization in a parallel environment is examined. Generating relation schemes in third normal form is straightforward when given a set of functional dependencies that is a reduced cover. It is shown that a reduced cover for a set of functional dependencies can be produced in parallel. The correctness of the algorithm is based on two important theorems. it is demonstrated that the companion third normal form algorithm can be easily translated into a parallel version. The performance of the two algorithms is compared to the performance of their serial counterparts. The standard serial algorithms for computing minimal covers and synthesizing third normal form relations are presented. The parallel algorithms and their rationale are discussed.>

The authors compare the performance of two join algorithms on both cube and ring interconnections for message-based multicomputers, and investigate the effects that the number of processors and the type of interconnection scheme have on the performance. First, the parallel hybrid-hash join algorithm and the parallel join-index join algorithm for both the cube and ring connected multicomputers are presented. The performance of these algorithms is then compared through analytical cost modeling. The result shows that the join-index join algorithm gives good performance only when the join selectivity is very small, and the hybrid-hash join algorithm performs consistently well under most situations. It is shown that the cube topology yields better execution time than the same algorithm on the ring topology. Furthermore, increasing the number of processors has a more significant improvement on the execution time of the cube than for the ring configuration. The applicability of join indexes on the parallel database algorithms is also discussed.>

The author examines join processing when the access paths available are nonclustered indexes on the joining attribute(s) for both relations involved in the join. He uses a bipartite graph model to represent the pages from the two relations that contain tuples to be joined. The minimization of the number of page accesses needed to compute a join in the author's database environment is explored from two perspectives. The first is to reduce the maximum buffer size so that no page is accessed more than once, and the second is to reduce the number of page accesses for a fixed buffer size. The author has developed heuristics for these problems. He gives performance comparisons of these heuristics and another method that recently appeared in the literature. Results show that one particular heuristic performs very well for addressing the problem from either perspective.>

Shows that the efficient reorganization of a B+ tree file into a linear hash file can be done concurrently with user transaction processing. This conversion is motivated by a change in database processing, in which efficient sequential and direct access were originally needed, but now only efficient direct access is needed. This is quite reasonable for a database system which accommodates new and changing applications. Several existing database systems, e.g. INGRES, IMS and IDMS allow this type of reorganization, but the reorganization is performed offline. The author devises an algorithm which performs the conversion and presents an analytic model of the conversion process. The author uses a typical database simulation model to evaluate the reorganization scheme. The results from the analytic model are within 3% (on the average) of the observed simulation results.>