The power-law approximation for the frequency distribution of words postulated by Zipf has been extensively studied for decades, which led to many variations on the theme. However, comparatively less attention has been paid to the investigation of the case of word frequencies. In this paper, we derive its analytical expression from the inverse of the underlying rank-size distribution as a function of total word count, vocabulary size and the shape parameter, thereby providing a unified framework to explain the nonlinear behavior of low frequencies on the log-log scale. We also present an efficient method based on relative entropy minimization for a robust estimation of the shape parameter using a small number of empirical low-frequency probabilities. Experiments were carried out for a selected set of languages with varying degrees of inflection in order to demonstrate the effectiveness of the proposed approach.

Compressing posting lists in large-scale search engines for efficiency improvement conventionally involves encoding document identifiers and term frequencies separately. In this work, we adopt dense unary codes in favor of joint encoding and evaluate them in terms of compression ratio along with two other techniques on eight document collections having different characteristics as to term frequency distribution. As a result, it has been observed that dense unary codes yield higher compression ratios particularly in short-text collections, where it is more unlikely for sparse terms to have high frequency values per document, as well as retaining a simple decoding mechanism.