In the current Big Data era, systems for collecting, storing and efficiently exploiting huge amounts of data are continually introduced, such as Hadoop, Apache Spark, Dremel, etc. Druid is one of theses systems especially designed to manage such data quantities, and allows to perform detailed real-time analysis on terabytes of data within sub-second latencies. One of the important Druid's requirements is fast data filtering. To insure that, Druid makes an extensive use of bitmap indexes. Previously, we introduced a new compressed bitmap index scheme called Roaring bitmap that has shown interesting results when compared to the bitmap compression scheme adopted by Druid: Concise. Since, Roaring bitmap has been integrated to Druid as an indexing solution. In this work, we produce an extensive series of experiments in order to compare Roaring bitmap and Concise time-space performances when used to accelerate Druid's OLAP queries and other kinds of operations Druid realizes on bitmaps, like: retrieving set bits from bitmaps, computing bitmap complements, aggregating several bitmaps with logical ORs and ANDs operations. Roaring bitmap has shown to improve up to ≈ 5× analytical queries response times under Druid compared to Concise.

Summary Bitmap indexes are commonly used in databases and search engines. By exploiting bit‐level parallelism, they can significantly accelerate queries. However, they can use much memory, and thus, we might prefer compressed bitmap indexes. Following Oracle's lead, bitmaps are often compressed using run‐length encoding (RLE). Building on prior work, we introduce the Roaring compressed bitmap format: it uses packed arrays for compression instead of RLE. We compare it to two high‐performance RLE‐based bitmap encoding techniques: Word Aligned Hybrid compression scheme and Compressed ‘n’ Composable Integer Set. On synthetic and real data, we find that Roaring bitmaps (1) often compress significantly better (e.g., 2×) and (2) are faster than the compressed alternatives (up to 900× faster for intersections). Our results challenge the view that RLE‐based bitmap compression is best. Copyright © 2015 John Wiley & Sons, Ltd.

Neural network is an important paradigm that has received little attention from the community of researchers in information retrieval, especially the auto-associative neural networks. These networks are capable of discovering patterns of terms among documents. We propose an auto-associative neural network to model the classification and to perform the matching task The unique layer network is trained with the documents of the collection and then used to recall the most relevant documents to specific queries. Our model has been tested on a TREC sub-collection. The results are compared against the vector space model, The experiment shows higher level of global precision and recall. The recall-precision curves show an important improvement on the precisions for the low levels of recall, which indicates a faster retrieval of the first relevant documents. This strength of the auto-associative neural network makes it an attractive model in information retrieval for general collections.

Information retrieval is concerned with the classification processes and the selective recovery of information. Improvements in this field are mainly sought at the core level of the engine's classification capabilities and by query enhancement processes. The later one became the prime interest of researchers since less progress has been made on the former one. Both make substantial use of manual interventions, which results in a less automated overall process. In this paper, we propose a new model based on the self-organizing map paradigm to discover the concepts embedded in a collection of documents. The terms of the corpus are directly classified into concepts, without manual category labelling. Then the concepts serve as a new knowledge representation for information retrieval. This model has been tested on a TREC-6 subcollection (text retrieval conference). As expected, the retrieval using the concepts representation does not outperform the corresponding full term retrieval. It is a step toward terms classification using a self-organizing map and contributes to fully automate the discovery of concepts in text collections.

Galois (concept) lattice theory has been successfully applied in data mining for the resolution of the association rule problem. In particular, structural results about lattices have been used in the design of efficient procedures for mining the frequent patterns (itemsets) in transaction databases. Since such databases are often dynamic, we propose a detailed study of the incremental aspects in lattice construction to support effective procedures for incremental mining of frequent closed itemsets (FCIs). Based on a set of descriptive results about lattice substructures involved in incremental updates, the paper presents a novel algorithm for lattice construction that explores only limited parts of a lattice for updating. Two new methods for incremental FCI mining are studied: the first inherits its extensive search strategy from a classical lattice method, whereas the second applies the new lattice construction strategy to the itemset mining context. Unlike batch techniques based on FCIs, both methods avoid rebuilding the FCI family from scratch whenever new transactions are added to the database and/or when the minimal support is changed.

During the evolution of object-oriented (OO) systems, the preservation of a correct design should be a permanent quest. However, for systems involving a large number of classes and that are subject to frequent modifications, the detection and correction of design flaws may be a complex and resource-consuming task. The use of automatic detection and correction tools can be helpful for this task. Various works have proposed transformations that improve the quality of an OO system while preserving its behavior. In this paper, we investigate whether some OO metrics can be used as indicators for automatically detecting situations where a particular transformation can be applied to improve the quality of a system. The detection process is based on analyzing the impact of various transformations on these OO metrics using quality estimation models.

During the evolution of object-oriented (OO) systems, the preservation of correct design should be a permanent quest. However, for systems involving a large number of classes and which are subject to frequent modifications, the detection and correction of design flaws may be a complex and resource-consuming task. Automating the detection and correction of design flaws is a good solution to this problem. Various authors have proposed transformations that improve the quality of an OO system while preserving its behavior. In this paper, we propose a technique for automatically detecting situations where a particular transformation can be applied to improve the quality of a system. The detection process is based on analyzing the impact of various transformations on software metrics using quality estimation models.

Learning concepts and rules from structured (complex) objects is a quite challenging but very relevant problem in the area of machine learning and knowledge discovery. In order to take into account and exploit the semantic relationships that hold between atomic components of structured objects, we propose a knowledge discovery process, which starts from a set of complex objects to produce a set of related atomic objects (called contexts). The second step of the process makes use of the concatenation product to get a global context in which binary relations of individual contexts coexist with relations produced by the application of some operators to individual contexts. The last step permits the discovery of concepts and implication rules using the concept lattice as a framework in order to discover and interpret nontrivial concepts and rules that may relate different components of complex objects. This paper focuses on two main steps of the knowledge discovery process, namely data mining and interpretation.

The Galois (or concept) lattice produced from a binary relation has proved useful for many applications. Building the Galois lattice can be considered a conceptual clustering method because it results in a concept hierarchy. This article presents incremental algorithms for updating the Galois lattice and corresponding graph, resulting in an incremental concept formation method. Different strategies are considered based on a characterization of the modifications implied by such an update. Results of empirical tests are given in order to compare the performance of the incremental algorithms to three other batch algorithms. Surprisingly, when the total time for incremental generation is used, the simplest and less efficient variant of the incremental algorithms outperforms the batch algorithms in most cases. When only the incremental update time is used, the incremental algorithm outperforms all the batch algorithms. Empirical evidence shows that, on the average, the incremental update is done in time proportional to the number of instances previously treated. Although the worst case is exponential, when there is a fixed upper bound on the number of features related to an instance, which is usually the case in practical applications, the worst‐case analysis of the algorithm also shows linear growth with respect to the number of instances.

This paper describes an approach to software reuse that involves generating and retrieving abstractions from existing software systems using concept formation methods. The potential of the approach is illustrated through two important activities of the reuse process. First, the concept hierarchy generated by the concept formation methods is used for organizing and retrieving the artifacts inside a repository. Second, the generated concepts are used in identifying new abstractions that may be converted into new, more generic artifacts with better reuse potential. These experiments are part of a major software engineering research project involving many business and academic partners.

An important structuring mechanism for knowledge bases is building an inheritance hierarchy of classes based on the content of their knowledge objects. This hierarchy facilitates group-related processing tasks such as answering set queries, discriminating between objects, finding similarities among objects, etc. Building this hierarchy is a difficult task for the knowledge engineer. Conceptual clustering may be used to automate or assist the engineer in the creation of such a classification structure. This article introduces a new conceptual clustering method which addresses the problem of clustering large amounts of structured objects. The conditions under which the method is applicable are discussed.>

Existing 00 development methodologies (see e.g.[Rumbaugh9 1 a, Wirfs-Brock90a, Coad9 la]) prescribe that design builds on the basic class structure identified at the analysis level--one aspect of the much vaunted seamless transition--by adding high level (control) and low level (utility)

An incremental algorithm for updating the Galois lattice is proposed where new objects may be dynamically added by modifying the existing lattice. A large experimental application reveals that adding a new object may be done in time proportional to the number of objects on the average. When there is a fixed upper bound on the number of properties related to an object, which is the case in practical applications, the worst case analysis of the algorithm confirms the experimental observations of linear growth with respect to the number of objects. Algorithms for generating rules from the lattice are also given.>

The need to add an automatic learning phase to the construction process of a knowledge base is stressed. This work introduces a new technique based on machine learning methodologies which automatically creates a particular knowledge representation structure called knowledge space, from which common generalizations of knowledge objects may be efficiently inferred. Also introduced are some of the improved and added processing capabilities made possible by this structure.>

In conventional Boolean retrieval systems, users have difficulty controlling the amount of output obtained from a given query. This paper describes the design of a user interface which permits gradual enlargement or refinement of the user's query by browsing through a graph of term and document subsets. This graph is obtained from a lattice automatically generated from the usual document-term relation. The major design features of the proposed interface are the integration of menu, fill-in the blank and direct manipulation modes of interaction within the “fisheye view” [Furnas, 1986] paradigm. A prototype user interface incorporating some of these ideas has been implemented on a microcomputer.