JPEG is a lossy compression technique which is the default format for all major digital cameras and estimated to be used for over 80% of all digital images on the web [1], [2]. During JPEG encoding, quantization is used to approximate frequency coefficients by a set of integers called the quantization table. We introduce OptQT as a new framework for the optimal design of JPEG quantization tables.

In this paper, we address the limitations of current measures for estimating the lossless compression limits. Shannon entropy, while practical, assumes a known data distribution and does not account for the complexity of representing this distribution. Kolmogorov Complexity (KC) [2], on the other hand, offers a more complete measure by considering both data and model complexity, but it is uncomputable in practice. To bridge these gaps, we propose a novel framework that estimates lower and upper bounds for lossless compression limits, leveraging neural scaling laws [1] to balance model and data complexity. Our experiments demonstrate the accuracy of our approach on synthetic datasets, with an average estimation error of 1.18%, and highlight its effectiveness as a tool for evaluating lossless compression methods on real-world datasets.

We propose ICE, an efficient and intelligent system that utilizes our dual-algorithm strategy to achieve superior lossless block-level compression results compared to traditional single compressor methods, especially with intermediate compression levels within the supported range of compressors. ICE employs a multi-step architecture to sample, analyze, plan, and execute compression. Unlike many existing solutions, ICE enables the user to set lower bounds (referred to as ‘PIVOT’) on throughput (compression speed) and compression ratio based on real-world constraints. While meeting these bounds, ICE maximizes the user's desired dimension. Our experimental results over several datasets from different domains demonstrate that ICE can improve the total compression throughput and compression ratio of these datasets with an average of 84% and 1.3% respectively, based on the selected PIVOT value.

JPEG is a widely used format for images. Most JPEG variants are based upon a block-based DCT transformation followed by quantization and entropy coding. Redundancy at row/column level is explored in [1]. Brunsli [2] and Lepton [3], lossless JPEG repacking libraries, explore redundancy at block level.

In the recent few years, lossless compressors based on deep learning emerged as a new research direction. However, existing DL compressors focus mostly on compression ratio at the expense of compression speed (typically 10-1000 times slower than popular compressors, operating at the KB/s level), rendering them impractical to be deployed in industrial settings. To address the throughput issue of network inference, we propose to utilize model compression in DL-based lossless data compression. Our approach originates from the sparse learning framework to flexibly balance the model complexity and the compression ratio. Through systematic numerical experiments, we found that compressed models with only tens of thousands of parameters can still retain competitive compression ratios comparing with the original large models which are typically 100 times larger. The proposed scheme reveals novel properties concerning deep compression models and data–logarithmic relations between the model size and compressed data size for general mixed data, and a turning point indicating minimum representation size for relatively formatted data.

A universal scheme is proposed for the lossless compression of two-dimensional tables and matrices. Instead of standard row- or column-based compression, we propose to sort each column first and record both the sorted table and the corresponding permutation table of the sorting permutations. These two tables are then separately compressed. In this new scheme, both intra- and inter-column correlations can be efficiently captured, giving rise to improved compression ratio in particular when both column-wise and row-wise dependencies cooccur. This scheme reduces the problem of the compression of an arbitrary two-dimensional table to that of a ‘permutation table’ together with a ‘sorted table’, where the former is only dependent on the table dimension and the latter can be effectively compressed column-by-column using predictive methods. Based on this scheme, a new algorithm is proposed, SortComp (sort-and-compress). For correlated columns, we give an estimation of the asymptotic bit rate of the algorithm and compare it to column-oriented compression schemes. Numerical experiments on real-life csv datasets validate the advantages of SortComp compared to existing row- and column-oriented compression algorithms.

Remote sensing are widely used in applications including geoexploration, topographic mapping and weather forecasting, producing vast amounts of multi and hyper-spectral image data that need to be compressed [1]. The data acquisition process often leads to artifacts in the form of stripes with unpredictable positions and amplitudes [2]. The stripes deteriorate the smoonthless of the original image, causing challenges for high-ratio lossless compression. This motivates us to propose a split-and-compress framework. Rather than direct compression, we split (decompose) the image into a smooth part and a sparse remainder (capturing the stripes and artifacts alike) and compress the two parts separately. The decomposition is achieved using a fast, robust statistics based method with linear computational complexity on the number of pixels.

Post-click conversion rate (CVR) prediction is an essential task for discovering user interests and increasing platform revenues in a range of industrial applications. One of the most challenging problems of this task is the existence of severe selection bias caused by the inherent self-selection behavior of users and the item selection process of systems. Currently, doubly robust (DR) learning approaches achieve the state-of-the-art performance for debiasing CVR prediction. However, in this paper, by theoretically analyzing the bias, variance and generalization bounds of DR methods, we find that existing DR approaches may have poor generalization caused by inaccurate estimation of propensity scores and imputation errors, which often occur in practice. Motivated by such analysis, we propose a generalized learning framework that not only unifies existing DR methods, but also provides a valuable opportunity to develop a series of new debiasing techniques to accommodate different application scenarios. Based on the framework, we propose two new DR methods, namely DR-BIAS and DR-MSE. DR-BIAS directly controls the bias of DR loss, while DR-MSE balances the bias and variance flexibly, which achieves better generalization performance. In addition, we propose a novel tri-level joint learning optimization method for DR-MSE in CVR prediction, and an efficient training algorithm correspondingly. We conduct extensive experiments on both real-world and semi-synthetic datasets, which validate the effectiveness of our proposed methods.