2022) is a BWT-based compressed index for computing the matching statistics and maximal exact matches (MEMs) of a pattern (usually a DNA read) with respect to a highly repetitive text (usually a database of genomes) using two operations: LF-steps and longest common extension (LCE) queries on a grammar-compressed representation of the text. In practice, most of the operations are constant-time LF-steps but most of the time is spent evaluating LCE queries. In this paper we show how (a variant of) the latter can be evaluated lazily, so as to bound the total time MONI needs to process the pattern in terms of the number of MEMs between the pattern and the text, while maintaining logarithmic latency.

The FM-index is one of the main success stories of the field of compact data structures and is a key part of many important tools in bioinformatics. Its primary weakness is a lack of access locality, with each step in a backward search typically causing several cache misses. If the indexed text is more than about lg σ times the size of cache, where σ is the size of the alphabet, then the bitvector at each level of the wavelet tree over the Burrows-Wheeler Transform (BWT) of the text may by itself be larger than cache — causing a cache miss as we descend from each level of the wavelet tree to the next. The resulting slowdown can be enough to cause practitioners to switch from FM-indexes to compressed suffix arrays, which have somewhat better locality.

Matching statistics were introduced to solve the approximate string matching problem, which is a recurrent subroutine in bioinformatics applications. In 2010, Ohlebusch et al. [SPIRE 2010] proposed a time and space efficient algorithm for computing matching statistics which relies on some components of a compressed suffix tree - notably, the longest common prefix (LCP) array. In this paper, we show how their algorithm can be generalized from strings to Wheeler deterministic finite automata. Most importantly, we introduce a notion of LCP array for Wheeler automata, thus establishing a first clear step towards extending (compressed) suffix tree functionalities to labeled graphs.

MONI (Rossi et al., 2022) can store a pangenomic dataset T in small space and later, given a pattern P, quickly find the maximal exact matches (MEMs) of P with respect to T. In this paper we consider its one-pass version (Boucher et al., 2021), whose query times are dominated in our experiments by longest common extension (LCE) queries. We show how a small modification lets us avoid most of these queries which significantly speeds up MONI in practice while only slightly increasing its size.

Until recently, most experts would probably have said we cannot backwards-step in constant time with a run-length compressed Burrows-Wheeler Transform (RLBWT), since doing so relies on rank queries on sparse bitvectors and those inherit lower bounds from predecessor queries. At ICALP'21, however, Nishimoto and Tabei [1] described a new, simple and constant-time implementation.

Suppose we want to store a string$S[1..n]$over an alphabet of size$\sigma$using adaptive prefix-free coding with fast encoding and decoding. If we are not too concerned about compression, we can process$/\mathrm{S}$in blocks of$\sigma$characters as follows: we encode or decode the first block with a Shannon code for the uniform distribution; to encode or decode the$ith$block, for$i > 1$, we build a Shannon code for the average of the distribution of characters we have seen so far and of the uniform distribution; since this averaged distribution assigns each character probability at least$1/(2\sigma)$(even when we have not seen it), the code-tree has height at most$[\text{lg}(2\sigma)] < \text{lg}\sigma+2$and can be represented with an$O(\sigma)$-space table that takes$O(\sigma)$time to build and allows constant-time encoding and decoding of characters - so we process the block in constant time per character. We store the first$\sigma$copies of each distinct character using fewer than$\text{lg}\sigma+2$bits each and, for$j > \sigma$, we store the$j\text{th}$copy$S[k]$using at most$\left[\text{lg}(\frac{1}{2}(\frac{j-\sigma+1}{\sigma\lfloor(k-1)/\sigma\rfloor}+\frac{1}{\sigma}))^{-1}\right]< \text{lg}\frac{k}{j-\sigma+1}+2$bits. Fairly straight forward calculation shows that when$\sigma\ll n$, we store at most$n(H+2)+o(n)$bits overall, where$H$is the entropy of the distribution of characters in$S$.

Generating pangenomic datasets is becoming increasingly common but there are still few tools able to handle them and even fewer accessible to non-specialists. Building compressed suffix trees (CSTs) for pangenomic datasets is still a major challenge but could be enormously beneficial to the community. In this paper, we present a method, which we refer to as RePFP-CST, for building CSTs in a manner that is scalable. To accomplish this, we show how to build a CST directly from VCF files without decompressing them, and to prune from the prefix-free parse (PFP) phrase boundaries whose removal reduces the total size of the dictionary and the parse. We show that these improvements reduce the time and space required for the construction of the CST, and the memory footprint of the finished CST, enabling us to build a CST for a terabyte of DNA for the first time in the literature.

As nowadays Machine Learning (ML) techniques are generating huge data collections, the problem of how to efficiently engineer their storage and operations is becoming of paramount importance. In this article we propose a new lossless compression scheme for real-valued matrices which achieves efficient performance in terms of compression ratio and time for linear-algebra operations. Experiments show that, as a compressor, our tool is clearly superior to gzip and it is usually within 20% of xz in terms of compression ratio. In addition, our compressed format supports matrix-vector multiplications in time and space proportional to the size of the compressed representation, unlike gzip and xz that require the full decompression of the compressed matrix. To our knowledge our lossless compressor is the first one achieving time and space complexities which match the theoretical limit expressed by the k -th order statistical entropy of the input. To achieve further time/space reductions, we propose column-reordering algorithms hinging on a novel column-similarity score. Our experiments on various data sets of ML matrices show that our column reordering can yield a further reduction of up to 16% in the peak memory usage during matrix-vector multiplication. Finally, we compare our proposal against the state-of-the-art Compressed Linear Algebra (CLA) approach showing that ours runs always at least twice faster (in a multi-thread setting), and achieves better compressed space occupancy and peak memory usage. This experimentally confirms the provably effective theoretical bounds we show for our compressed-matrix approach.

Computing the matching statistics of patterns with respect to a text is a fundamental task in bioinformatics, but a formidable one when the text is a highly compressed genomic database. Bannai et al. gave an efficient solution for this case, which Rossi et al. recently implemented, but it uses two passes over the patterns and buffers a pointer for each character during the first pass. In this paper, we simplify their solution and make it streaming, at the cost of slowing it down slightly. This means that, first, we can compute the matching statistics of several long patterns (such as whole human chromosomes) in parallel while still using a reasonable amount of RAM; second, we can compute matching statistics online with low latency and thus quickly recognize when a pattern becomes incompressible relative to the database. Our code is available at https://github.com/koeppl/phoni.

Large sequencing projects, such as GenomeTrakr and MetaSub, are updated frequently (sometimes daily, in the case of GenomeTrakr) with new data. Therefore, it is imperative that any data structure indexing such data supports efficient updates. Toward this goal, Bannai et al. (TCS, 2020) proposed a data structure named dynamic r-index which is suitable for large genome collections and supports incremental construction; however, it is still not powerful enough to support substantial updates. Here, we develop a novel algorithm for updating the r-index, which we refer to as RIMERGE. Fundamental to our algorithm is the combination of the basics of the dynamic r-index with a known algorithm for merging Burrows-Wheeler Transforms (BWTs). As a result, RIMERGE is capable of performing batch updates in a manner that exploits parallelism while keeping the memory overhead small. We compare our method to the dynamic r-index of Bannai et al. using two different datasets, and show that RIMERGE is between 1.88 to 5.34 times faster on reasonably large inputs.

As the number of vehicles and devices equipped with GPS technology has grown explosively, an urgent need has arisen for time- and space-efficient data structures to represent their trajectories. The most commonly desired queries are the following: queries about an object’s trajectory, range queries, and nearest neighbor queries. In this paper, we consider that the objects can move freely and we present a new compressed data structure for storing their trajectories, based on a combination of logs and snapshots, with the logs storing sequences of the objects’ relative movements and the snapshots storing their absolute positions sampled at regular time intervals. We call our data structure ContaCT because it provides Constant- time access to Compressed Trajectories. Its logs are based on a compact partial-sums data structure that returns cumulative displacement in constant time, and allows us to compute in constant time any object’s position at any instant, enabling a speedup when processing several other queries. We have compared ContaCT experimentally with another compact data structure for trajectories, called GraCT, and with a classic spatio-temporal index, the MVR-tree. Our results show that ContaCT outperforms the MVR-tree by orders of magnitude in space and also outperforms the compressed representation in time performance.

We consider the problem of decompressing the Lempel-Ziv 77 representation of a string S of length n using a working space as close as possible to the size z of the input. The folklore solution for the problem runs in O(n) time but requires random access to the whole decompressed text. Another folklore solution is to convert LZ77 into a grammar of size O(z log(n/z)) and then stream S in linear time. In this paper, we show that O(n) time and O(z) working space can be achieved for constant-size alphabets. On general alphabets of size σ, we describe (i) a trade-off achieving O(n logδσ) time and O(z log1-δσ) space for any 0≤ δ≤ 1, and (ii) a solution achieving O(n) time and O(z log log (n/z)) space. The latter solution, in particular, dominates both folklore algorithms for the problem. Our solutions can, more generally, extract any specified subsequence of S with little overheads on top of the linear running time and working space. As an immediate corollary, we show that our techniques yield improved results for pattern matching problems on LZ77-compressed text.

Baier (CPM 2018) describes tunneling as a technique to further exploit redundancies in the Burrows-Wheeler Transform. In this paper we show how to retain indexed text searching on the resulting structure and generalize the concept to Wheeler graphs.

The Block Tree (BT) is a novel compact data structure designed to compress sequence collections. It obtains compression ratios close to Lempel-Ziv and supports efficient direct access to any substring. The BT divides the text recursively into fixed-size blocks and those appearing earlier are represented with pointers. On repetitive collections, a few blocks can represent all the others, and thus the BT reduces the size by orders of magnitude. In this paper we extend the BT to two dimensions, to exploit repetitiveness in collections of images, graphs, and maps. This two-dimensional Block Tree divides the image regularly into subimages and replaces some of them by pointers to other occurrences thereof. We develop a specific variant aimed at compressing the adjacency matrices of Web graphs, obtaining space reductions of up to 50% compared with the k2-tree, which is the best alternative supporting direct and reverse navigation in the graph.

Computing the product of the (binary) adjacency matrix of a large graph with a real-valued vector is an important operation that lies at the heart of various graph analysis tasks, such as computing PageRank. In this paper we show that some well-known Web and social graph compression formats are computation-friendly, in the sense that they allow boosting the computation. In particular, we show that the format of Boldi and Vigna allows computing the product in time proportional to the compressed graph size. Our experimental results show speedups of at least 2 on graphs that were compressed at least 5 times with respect to the original. We show that other successful graph compression formats enjoy this property as well.

Order-preserving pattern matching was first studied surprisingly recently buthas already attracted much attention. For this problem we propose aspace-efficient index that works well in practice despite its lack of goodworst-case time bounds. Our solution is based on the new approach ofdecomposing the indexed sequence into an em order component, containingordering information, and a δ component, containing informationon the absolute values. Experiments show that this approach is viable and itis the first one offering simultaneously small space usage and fast retrieval.

In the range α-majority query problem, we preprocess a given sequence S[1..n] for a fixed threshold α ∈ (0, 1], such that given a query range [i..j], the symbols that occur more than α (j-i+1) times in S[i..j] can be reported efficiently. We design the first compressed solution to this problem in dynamic settings. Our data structure represents S using nHko(nlg σ) bits for any k = o(log σ n), where σ is the alphabet size and Hkis the k-th order empirical entropy of S. It answers range α-majority queries in O((lg n)/(α lg lgn)) time, and supports insertions and deletions in O(lg n/α) amortized time. The best previous solution [1] has the same query and update times, but uses O(n) words.

Most of the fastest-growing string collections today are repetitive, that is, most of the constituent documents are similar to many others. As these collections keep growing, a key approach to handling them is to exploit their repetitiveness, which can reduce their space usage by orders of magnitude. We study the problem of indexing repetitive string collections in order to perform efficient document retrieval operations on them. Document retrieval problems are routinely solved by search engines on large natural language collections, but the techniques are less developed on generic string collections. The case of repetitive string collections is even less understood, and there are very few existing solutions. We develop two novel ideas, interleaved LCPs and precomputed document lists, that yield highly compressed indexes solving the problem of document listing (find all the documents where a string appears), top-k document retrieval (find the k documents where a string appears most often), and document counting (count the number of documents where a string appears). We also show that a classical data structure supporting the latter query becomes highly compressible on repetitive data. Finally, we show how the tools we developed can be combined to solve ranked conjunctive and disjunctive multi-term queries under the simple $${\textsf{tf}}{\textsf{-}}{\textsf{idf}}$$ model of relevance. We thoroughly evaluate the resulting techniques in various real-life repetitiveness scenarios, and recommend the best choices for each case.

We describe a data structure that stores a strings in space similar to that of its Lempel-Ziv encoding and efficiently supports access, rank and select queries. These queries are fundamental for implementing succinct and compressed data structures, such as compressed trees and graphs. We show that our data structure can be built in a scalable manner and is both small and fast in practice compared to other data structures supporting such queries.

The de Bruijn graph GK of a set of strings Sis a key data structure in genome assembly that represents overlaps between all the K-length substrings of S. Construction and navigation of the graph is a space and time bottleneck in practice and the main hurdle for assembling large genomes. This problem is compounded because state-of-the-art assemblers do not build the de Bruijn graph for a single order (value of K) but for multiple values of K: they builddde Bruijn graphs, each with a specific order, i.e., GK1, GK2, GKd. Al-though, this paradigm increases the quality of the assembly produce but it greatly increases runtime, because of the need to construct graphs instead of one. In this paper, we show how to augment a succinct de Bruijn graph representation by Bowe et al. (Proc. WABI, 2012) to support new operations that let us change order on the fly, effectively representing all de Bruijn graphs of order up to some maximum Kin a single data structure. Our experiments show our variable-order de Bruijn graph only modestly increases space usage, construction time, and navigation time compared to a single order graph.

Real-world point sets tend to be clustered, so using a machine word for each point is wasteful. In this paper we first bound the number of nodes in the quad tree for a point set in terms of the points' clustering. We then describe aqua tree data structure that uses O (1) bits per node and supports faster queries than previous structures with this property. Finally, we present experimental evidence that our structure is practical.

We address the problem of counting the number of strings in a collection where a given pattern appears, which has applications in information retrieval and data mining. Existing solutions are in a theoretical stage. In this pa-per we implement these solutions and explore compressed variants, aiming to reduce data structure size. Our main result is to uncover some unexpected compressibility properties of the fastest known data structure for the problem. By taking advantage of these properties, we can reduce the size of the structure by a factor of 5-400, depending on the dataset.

Relative Lempel-Ziv [1] (RLZ) is a variant of LZ77 that can compress collections of similar genomes well, while still allowing fast random access to them. We implemented RLZ using compressed bit vectors to support constant-time random access at the cost of sublinear extra space. We compared our implementation of RLZ to Deorowicz and Grabowski's GDC [11] scheme and achieved comparable compression and much smaller access times for short substrings.

In this paper we study the adaptive prefix coding problem in cases where the size of the input alphabet is large. We present an online prefix coding algorithm that uses O(sigma1/lambda+epsiv) bits of space for any constants epsiv > 0, > 1, and encodes the string of symbols in O(loglog sigma) time per symbol in the worst case, where sigma is the size of the alphabet. The upper bound on the encoding length is lambdanH(s) + (lambda/ ln 2 + 2 + epsiv)n + O(sigma1/lambdalog2sigma) bits.

Summary form only given. Internet users usually download more than they upload and many technologies have asymmetric bandwidth. Suppose some clients want to send messages to a server. At any point, the server knows all the messages it has received so far; each client only knows its own message or messages and does not overhear communication between other clients and the server. Thus, the server may be able to compress the messages but the clients individually cannot. By assuming the server, after receiving a sample of messages, can accurately estimate the distribution of all the messages, a multi-round asymmetric communication protocol can be provided in which the server uses its greater bandwidth to help a single client send a message drawn from a distribution known to the server. The server can just repeat their protocol to help any number of clients, provided it starts with a representative sample. This paper propose a new protocol inspired by an everyday act: placing a call on a cell phone. While the protocol needs no assumptions and is nearly optimal with respect to the number of bits the clients send, the server sends a relatively large number of bits, which is necessary in order to use only one round of communication for each message

This paper presents a new algorithm, called dynamic Shannon coding, that uses at most (H + 1)m + O(nlogm) bits to encode string S. The key idea is to smooth the relative frequencies of characters when computing their weights. It also shows that dynamic Shannon coding can be easily modified to restrict the maximum length of any codeword in the encoding produced. The analysis of dynamic Shannon coding is much simpler than the analysis of dynamic Huffman coding.