We study the problem of compressing massive tables within the partition-training paradigm introduced by Buchsbaum et al. [2000], in which a table is partitioned by an off-line training procedure into disjoint intervals of columns, each of which is compressed separately by a standard, on-line compressor like gzip. We provide a new theory that unifies previous experimental observations on partitioning and heuristic observations on column permutation, all of which are used to improve compression rates. Based on this theory, we devise the first on-line training algorithms for table compression, which can be applied to individual files, not just continuously operating sources; and also a new, off-line training algorithm, based on a link to the asymmetric traveling salesman problem, which improves on prior work by rearranging columns prior to partitioning. We demonstrate these results experimentally. On various test files, the on-line algorithms provide 35--55% improvement over gzip with negligible slowdown; the off-line reordering provides up to 20% further improvement over partitioning alone. We also show that a variation of the table compression problem is MAX-SNP hard.

Programmers frequently have to retrieve and link information from various software documents to accomplish a maintenance task. Ciao is a graph-based navigator that helps programmers query and browse structural connections embedded in different software and document repositories. A repository consists of a collection of source documents with an associated database that describes their structure. Ciao supports repositories organized in an architecture style called Aero, which exploits the duality between a class of entity-relationship (ER) databases and directed attributed graphs (DAG). Database queries and graph analysis operators in Aero are plug-compatible because they all take an ER database and produce yet another ER database by default. Various presentation filters generate graph views, source views, and relational views from any compatible ER database. The architecture promotes the construction of successively more complex operators using a notion of virtual database pipelines. Ciao has been instantiated for C and C++ program databases, and program difference databases. The latter allows programmers to explore program structure changes by browsing and expanding graphs that highlight changed, deleted, and added entities and relationships. The unifying ER model under ciao also allows users to navigate different software repositories and make necessary connections. We have linked program difference databases and modification request (MR) databases so that users can investigate the connections between MRs and affected entities. Ciao has been applied to several large communications software projects and we report experiences and lessons learned from these applications.