Ransomware is software that uses encryption to disable access to data until a ransom is paid and such attacks have increased steeply in recent times. The best current practice to minimize the impact of ransomware attacks include periodic backups and airgapped immutable copies. However, undetected attacks can corrupt data before backups, making backups unusable. Detecting ransomware attacks quickly and flagging the damaged content enables fast recovery and business continuity. We present some features of our ransomware attack detection algorithms prototyped and run on a sandboxed but realistic environment that successfully detected the live ransomware attacks from open source repositories.

Despite the large number of research papers and compression algorithms proposed for com- pressing FASTQ genomic data generated by sequencing machines, by far the most commonly used compression algorithm in the industry for FASTQ data is gzip. The main drawback of the proposed alternative special-purpose compression algorithms is the slow speed of either compression or de- compression or both, and also their brittleness by making various limiting assumptions about the input FASTQ format (for example, the structure of the headers or fixed lengths of the records [5]) in order to further improve their specialized compression. In this paper we propose using a simple modeling improvement combined with fast general purpose encoders, achieving 4X-40X speed improvements and 6%-11% more efficienct compression ratios for the compression and decompression of FASTQ genomic data compared to gzip and also being 5X faster than special-purpose FASTQ compressors like DSRC v2 and slimfastq.

Summary form only given. Primary storage systems that compress data in real time, use some form of on disk metadata to perform the virtualization needed in storing compressed data. Usually this metadata is in the form of B-trees (eventually compressed) and stored on disk. For random accesses to compressed data, where the metadata is not in cache, this additional layer significantly slows down random reads and writes. Our solution is to use much less metadata that only provides an approximation of the location of compressed data on disk and can be easily stored in the memory of the storage system. Read operations are extended to compensate for the imprecise position information in the metadata, and index marks embedded in the data are used to locate the required data within the expanded read. The data placement of written data is constrained to be described by the reduced metadata. The placement uses a piecewise linear scheme based on the locality in compressibility of data and we support this assumption with experiments.

Deduplication is a commonly-used technique on disk-based storage pools. However, deduplication has not been used for tape-based pools: tape characteristics, such as high mount and seek times combined with data fragmentation resulting from deduplication create a toxic combination that leads to unacceptably high retrieval times. This work proposes DedupT, a system that efficiently supports deduplication on tape pools. This paper (i) details the main challenges to enable efficient deduplication on tape libraries, (ii) presents a class of solutions based on graph-modeling of similarity between data items that enables efficient placement on tapes; and (iii) presents the design and evaluation of novel cross-tape and on-tape chunk placement algorithms that alleviate tape mount time overhead and reduce on-tape data fragmentation. Using 4.5 TB of real-world workloads, we show that DedupT retains at least 95% of the deduplication efficiency. We show that DedupT mitigates major retrieval time overheads, and, due to reading less data, is able to offer better restore performance compared to the case of restoring non-deduplicated data.

Modern primary storage systems support or intend to add support for real time compression usually based on some flavor of the LZ77 and/or Huffman algorithm. There is a fundamental tradeoff in adding real time (adaptive) compression to such a system: to get good compression the amount of compressed data (the independently compressed block) should be large, to be able to read quickly from random places the blocks should be small. One idea is to let the independently compressed blocks be large but to be able to start decompressing the needed part of the block from a random location inside the compressed block. We explore this idea and compare it with a few alternatives, experimenting with the \zlib\ code base.

There has been increasing interest in deploying data reduction techniques in primary storage systems. This paper analyzes large datasets in four typical enterprise data environments to find patterns that can suggest good design choices for such systems. The overall data reduction opportunity is evaluated for deduplication and compression, separately and combined, then in-depth analysis is presented focusing on frequency, clustering and other patterns in the collected data. The results suggest ways to enhance performance and reduce resource requirements and system cost while maintaining data reduction effectiveness. These techniques include deciding which files to compress based on file type and size, using duplication affinity to guide deployment decisions, and optimizing the detection and mapping of duplicate content adaptively when large segments account for most of the opportunity.

Many new storage systems provide some form of data reduction. We examine data reduction methods that might be suitable for \emph{primary} storage systems serving active data (as contrasted with backup and archive systems), by analysis of file sets found in different active data environments. We address questions of: how effective are compression and variations of deduplication, both separately and in combination, when deduplication and compression are combined, which should be applied first, what will the tradeoff be between the different methods in their use of MIPS relative to the data reduction achieved, and what degree of data reduction should be expected for different data types.

Data deduplication is a popular dictionary based compression method in storage archival and backup. The deduplication efficiency improves for smaller chunk sizes, however the files become highly fragmented requiring many disk accesses during reconstruction or chattiness in a client-server architecture. Within the sequence of chunks that an object (file) is decomposed into, sub-sequences of adjacent chunks tend to repeat. We exploit this insight to optimize the chunk sizes by joining repeated sub-sequences of small chunks into new super chunks with the constraint to achieve practically the same matching performance. We employ suffix arrays to find these repeating sub-sequences and to determine a new encoding that covers the original sequence. With super chunks we significantly reduce fragmentation, improving reconstruction time and the overall deduplication ratio by lowering the amount of metadata. As a result, fewer chunks are used to represent a file, reducing the number of disk accesses needed to reconstruct the file and requiring fewer entries in the chunk dictionary and fewer hashes to encode a file. To encode a sequence of chunks we proved two facts: (1) any subsequence that repeats is part of some super chunk (su- permaximal in Figure 1) - therefore the dictionary contains just supermaximals and non-repeats, and (2) maximals are the only repeats not covered (overlapped) by the containing supermaximals - so once we discover the maximals with the suffix array, and encode them, there is no need for maintaining auxiliary data structures like bit masks, to guarantee the coverage (encoding) of the entire object. Our experimental evaluation (Figure 2) yields a reduction in fragmentation between 89%-97% and a reduction of dictionary metadata (number of entries) between 80%-97%, without increasing the total amount of storage required for unique chunks.

Data de-duplication is a simple compression method, popular in storage archival and backup that consists in partitioning large data objects (files) into smaller parts (named chunks), and replacing the chunks for the purpose of communication or storage by their ID, generally a cryptographic hash like SHA-1 of the chunk data [A. Muthitacharoen et al., 2001], [D.R. Bobbarjung et al., 2006]. The compression ratio achieved by de-duplication can be improved by (1) increasing the likelihood of matching the new chunks against the dictionary (archived) chunks and/or (2) compressing the list of hashes (indexes, of 20 bytes each). Using smaller chunk sizes increases the chance of matching but many more hashes will be generated. The chunks repository is a hash table where each entry stores the SHA-1 value of the chunk and the chunk data. In addition, with each newly created entry we store a chronological pointer linking it with the next new entry. When the hashes produced by the chunker follow the chronological pointers we encode them as a sequence of hashes by specifying the first hash in the sequence and the length of the sequence or when the same hash is generated repeatedly we encode it as a run of hashes by specifying its value and the number of repeated occurrences. The usefulness of the chronological pointers is derived from the insight that when archiving successive versions of a file or set of files, large contiguous areas remain unchanged between these versions and the chronological pointers are predictors of this contiguity. If the contiguity is broken there is a small loss in the hash sequence compression.

Constantinescu and Storer presented a new single-pass adaptive vector quantization algorithm that learns a codebook of variable size and shape entries; they presented experiments on a set of test images showing that with no training or prior knowledge of the data, for a given fidelity, the compression achieved typically equals or exceeds that of the JPEG standard. This paper presents improvements in speed (by employing K-D trees), simplicity of codebook entries, and visual quality with no loss in either the amount of compression or the SNR as compared to the original full-search version.>