In probabilistic approaches to information retrieval, the occurrence of a query term in a document contributes to the probability that the document will be judged relevant. It is typically assumed that the weight assigned to a query term should be based on the expected value of that contribution. In this paper we show that the degree to which observable document features such as term frequencies are expected to vary is also important. By means of stochastic simulation, we show that increased variance results in degraded retrieval performance. We further show that by decreasing term weights in the presence of variance, this degradation can be reduced. Hence, probabilistic models of information retrieval must take into account not only the expected value of a query term's contribution but also the variance of document features.

This paper takes a fresh look at modeling approaches to information retrieval that have been the basis of much of the probabilistically motivated IR research over the last 20 years. We shall adopt a subjectivist Bayesian view of probabilities and argue that classical work on probabilistic retrieval is best understood from this perspective. The main focus of the paper will be the ranking formulas corresponding to the Binary Independence Model (BIM), presented originally by Roberston and Sparck Jones [1977] and the Combination Match Model (CMM), developed shortly thereafter by Croft and Harper [1979]. We will show how these same ranking formulas can result from a probabilistic methodology commonly known as Maximum Entropy (MAXENT).

Article A language modeling approach to information retrieval Share on Authors: Jay M. Ponte Computer Science Department, University of Massachusetts, Amherst Computer Science Department, University of Massachusetts, AmherstView Profile , W. Bruce Croft Computer Science Department, University of Massachusetts, Amherst Computer Science Department, University of Massachusetts, AmherstView Profile Authors Info & Claims SIGIR '98: Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrievalAugust 1998 Pages 275–281https://doi.org/10.1145/290941.291008Online:01 August 1998Publication History 1,592citation7,260DownloadsMetricsTotal Citations1,592Total Downloads7,260Last 12 Months258Last 6 weeks34 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access