The purpose of this half-day workshop is to bring together academic and industry interactive information retrieval (IIR) researchers with an interest in evaluation methodologies. The workshop articulates contemporary challenges in the investigation of IIR and invites user- and system-oriented researchers to work collaboratively to address these challenges by combining user- and system-centered methodologies in meaningful ways. We anticipate that this workshop will initiate productive knowledge exchange and partnerships that can respond to the increasing user, task, system, and contextual complexity of the IIR field.

The assumptions underlying the Probability Ranking Principle (PRP) have led to a number of alternative approaches that cater or compensate for the PRP's limitations. In this poster we focus on the Interactive PRP (iPRP), which rejects the assumption of independence between documents made by the PRP. Although the theoretical framework of the iPRP is appealing, no instantiation has been proposed and investigated. In this poster, we propose a possible instantiation of the principle, performing the first empirical comparison of the iPRP against the PRP. For document diversification, our results show that the iPRP is significantly better than the PRP, and comparable to or better than other methods such as Modern Portfolio Theory.

The probabilistic formalism of quantum physics is said to provide a sound basis for building a principled information retrieval framework. Such a framework can be based on the notion of information need vector spaces where events, such as document relevance or observed user interactions, correspond to subspaces. As in quantum theory, a probability distribution over these subspaces is defined through weighted sets of state vectors (density operators), and used to represent the current view of the retrieval system on the user information need. Tensor spaces can be used to capture different aspects of information needs. Our evaluation shows that the framework can lead to acceptable performance in an ad-hoc retrieval task. Going beyond this, we discuss the potential of the framework for three active challenges in information retrieval, namely, interaction, novelty and diversity.

The Quantum Probability Ranking Principle (QPRP) has been recently proposed, and accounts for interdependent document relevance when ranking. However, to be instantiated, the QPRP requires a method to approximate the "interference" between two documents. In this poster, we empirically evaluate a number of different methods of approximation on two TREC test collections for subtopic retrieval. It is shown that these approximations can lead to significantly better retrieval performance over the state of the art.

Recently, Portfolio Theory (PT) has been proposed for Information Retrieval. However, under non-trivial conditions PT violates the original Probability Ranking Principle (PRP). In this poster, we shall explore whether PT upholds a different ranking principle based on Quantum Theory, i.e. the Quantum Probability Ranking Principle (QPRP), and examine the relationship between this new model and the new ranking principle. We make a significant contribution to the theoretical development of PT and show that under certain circumstances PT upholds the QPRP, and thus guarantees an optimal ranking according to the QPRP. A practical implication of this finding is that the parameters of PT can be automatically estimated via the QPRP, instead of resorting to extensive parameter tuning.

Blogs facilitate online debates and discussions for millions of people around the world. Identifying the most popular and prevailing topics discussed in the Blogosphere is a crucial task. This poster describes our novel approach to the quantification of the level of topic propagation in the Blogosphere. Our model uses graph-theoretic representations of the Blogosphere's link structures that allows it to deduce the `Percolation Threshold', which is then used in the quantification and definition of a global topic. The result of our experiments on a blog collection shows that our model is able to quantify the propagation of topics. Moreover, our model is successful in identifying specific topics that propagate throughout the Blogosphere and classifies them as `Global'.

Retrieval with Logical Imaging is derived from belief revision and provides a novel mechanism for estimating the relevance of a document through logical implication (i.e. P(q->d). In this poster, we perform the first comprehensive evaluation of Logical Imaging (LI) in Information Retrieval (IR) across several TREC test Collections. When compared against standard baseline models, we show that LI fails to improve performance. This failure can be attributed to a nuance within the model that means non-relevant documents are promoted in the ranking, while relevant documents are demoted. This is an important contribution because it not only contextualizes the effectiveness of LI, but crucially explains why it fails. By addressing this nuance, future LI models could be significantly improved.

The ability to formally analyse and represent semantic relations of terms is a major challenge for many areas of computing science and an intriguing problem for other sciences. In applications of evidence theory to, for instance, information retrieval, the problem of analysis and representation becomes apparent because evidence theory is based on set theory and individual key terms have to be modelled as subsets of the frame of discernment. How to find the frame and model the key terms is a challenge. The problem leads to other practical problems, as pointed out repeatedly in the literature. In this study, we focus on such a problem, present a method for simplifying and normalizing a thesaurus, and propose an algorithm for establishing the frame of discernment and for modelling individual key terms as a subset of the frame. The key aim of this study is to treat semantic relations of terms by means of a normalized thesaurus. © 2008 Wiley Periodicals, Inc.

This acceptance talk is a curious mixture of personal history and developing ideas in the context of the growing field of IR covering several decades. I want to concentrate on models and theories, interpreted loosely, and try and give an insight into where I have got to in my thinking, where the ideas came from, and where I believe we are going.In the last few years I have been working on the development of what might be coined as a design language for IR. It takes its inspiration from Quantum Mechanics, but by analogy only. The mathematical objects represent documents; these objects might be vectors (or density operators) in an n-dimensional vector space (usually a Hilbert space). A request for information, or a query, is taken as an observable and is represented as a linear operator on the space. Linear operators can be expressed as matrices. Such an operator, Hermitian, has a set of eigenvectors forming a basis for the space; which we interpret as a point of view or perspective from which to understand the space. Thus any document-vector can be located with respect to the basis, and we can calculate an inner product between such a vector and any basis vector, which may be interpreted as a probability of relevance. The probability of observing any given eigenvector is now given by the square of that inner product assuming all vectors are normalised. Hence we connect the probability of observation to the geometry of the space. Furthermore, the subspaces of the space make up a lattice structure which is equivalent to a logic. This makes up the entire mathematical structure, and the language for handling this structure is linear algebra: vectors, matrices, projections, inner-products, neatly captured by the Dirac notation used in quantum mechanics. Our probability is slightly different from classical probability, the same for logic; we end up with quantum logic and quantum probability.A commitment to this kind of mathematical structure, with which to model objects and processes in IR, depends on two critical assumptions.

In this article we describe an evaluation of relevance feedback (RF) algorithms using searcher simulations. Since these algorithms select additional terms for query modification based on inferences made from searcher interaction, not on relevance information searchers explicitly provide (as in traditional RF), we refer to them as implicit feedback models . We introduce six different models that base their decisions on the interactions of searchers and use different approaches to rank query modification terms. The aim of this article is to determine which of these models should be used to assist searchers in the systems we develop. To evaluate these models we used searcher simulations that afforded us more control over the experimental conditions than experiments with human subjects and allowed complex interaction to be modeled without the need for costly human experimentation. The simulation-based evaluation methodology measures how well the models learn the distribution of terms across relevant documents (i.e., learn what information is relevant) and how well they improve search effectiveness (i.e., create effective search queries). Our findings show that an implicit feedback model based on Jeffrey's rule of conditioning outperformed other models under investigation.

We propose a topic based approach to language modelling for ad-hoc information retrieval (IR). Many smoothed estimators used for the multinomial query model in IR rely upon the estimated background collection probabilities. In this paper, we propose a topic based language modelling approach, that uses a more informative prior based on the topical content of a document. In our experiments, the proposed model provides comparable IR performance to the standard models, but when combined in a two stage language model, it outperforms all other estimated models.

Capitalizing on the intuitive underlying assumptions of Language Modelling for Ad-Hoc Retrieval we present a novel approach that is capable of injecting the user's context of the document collection into the retrieval process. The preliminary findings from the evaluation undertaken suggest that improved IR performance is possible under certain circumstances. This motivates further investigation to determine the extent and significance of this improved performance.

An empirical study has been conducted investigating the relationship between the performance of an aspect based language model in terms of perplexity and the corresponding information retrieval performance obtained. It is observed, on the corpora considered, that the perplexity of the language model has a systematic relationship with the achievable precision recall performance though it is not statistically significant.

Abstract Research on the use of mathematical, logical, and formal methods, has been central to Information Retrieval research for long time. Research in this area is important not only because it helps enhancing retrieval effectiveness, but also because it helps clarifying the underlying concepts of Information Retrieval. In this article we outline some of the major aspects of the subject, and summarize the papers of this special issue with respect to how they relate to these aspects. We conclude by highlighting some directions of future research, which are needed to better understand the formal characteristics of Information Retrieval.

Abstract In this paper, we present five user experiments on incorporating behavioral information into the relevance feedback process. In particular, we concentrate on ranking terms for query expansion and selecting new terms to add to the user's query. Our experiments are an attempt to widen the evidence used for relevance feedback from simply the relevant documents to include information on how users are searching. We show that this information can lead to more successful relevance feedback techniques. We also show that the presentation of relevance feedback to the user is important in the success of relevance feedback.

For many years I have wanted to give a talk like this: look back on our subject, identify the high (and perhaps low) points, consider what worked, what did not work, and speculate a little about the future. Now that I at last have the opportunity to give such a talk the realisation has dawned just how difficult it is to do justice to the topic. The only way out of this difficulty for me is to emphasise that this is a personal account, based on my involvement with the field since 1968, and that errors of omission and commission are not deliberate but simply due to lack of knowledge and time on my part.To talk of landmarks is easy but to say what they are in IR is not. They come in various shapes and sizes: events, publications, experiment, ideas, etc. In the course of this presentation I shall be judiciously mixing all of these. However, the emphasis will be on ideas and their subsequent modelling and testing through experimentation. The interaction between theory and experiment will be a recurring theme. I will try and associate these developments with key individuals, thereby running the risk of ignoring some; I apologise for this in advance.The pre-history of our subject can be traced back to the work in the 19th century, perhaps even further, but I will pick it up at the middle of the last century (20th) starting with the work of Robert Fairthorne and Vannevar Bush. This early work emphasised the possibility of using mechanical devices to store and retrieve information. Of course the foundations of modern information retrieval were properly laid after 1945 with the pioneering work of Cleverdon, Salton, Sparck Jones, and others. This work gave rise to a strong experimental methodology for the evaluation of theoretical ideas, which has been sustained to this day. It has been a hallmark of IR research that theory is developed in the context of experimentation. There is no doubt that many disciplines are jealous of the success of TREC.IR research has thrown up a number of successful models. These models have been based on some, often unstated, assumptions (or hypotheses). I will attempt to identify some of the underlying ideas, giving credit where is due, that led to the fruitful exploration of retrieval models. This will include system-oriented as well as user-oriented ideas, especially those concerned with the measurement of retrieval performance.IR has been fortunate in that the subject has grown through the active collaboration between computer scientists and information scientists. This has meant that traditional approaches to the storage and retrieval of information emanating from the library world, for example, have always strongly influenced new developments. This tension between manual (human) processes and automatic computer-based processes in IR has always been fruitful. Even now with the evolution of ideas about meta-data and ontologies needed to enhance web retrieval, the debate about controlled vocabularies versus automatic indexing is relevant. Issues of scalability are particularly important here.One of the strengths that have emerged in our subject is that many of our models can be deployed independently of medium or modality. For example, retrieving images or audio sequences can be handled in ways similar to those used to retrieve text data. This has proved to be great boon to IR. The development of web retrieval through the deployment of various kinds of search engines has been based on the considerable early work in IR although detailing the specific influences is not easy. It is clear that the underlying mathematical and statistical models in IR have been ubiquitous in application. The extreme difficulty encountered in making NLP work for IR forced researchers to develop powerful statistical, probabilistic, geometrical, and logical techniques to complement linguistic ones. This is now paying off because of the similar difficulties encountered in other media.Having given some account of how we got here I will spend a little time talking about where we go from here, how do we extract the message from the bottle?

Abstract In this article we report on a series of experiments designed to investigate the combination of term and document weighting functions in information retrieval. We describe a series of weighting functions, each of which is based on how information is used within documents and collections, and use these weighting functions in two types of experiments: one based on combination of evidence for ad hoc retrieval, the other based on selective combination of evidence within a relevance feedback situation. We discuss the difficulties involved in predicting good combinations of evidence for ad hoc retrieval, and suggest the factors that may lead to the success or failure of combination. We also demonstrate how, in a relevance feedback situation, the relevance assessments can provide a good indication of how evidence should be selected for query term weighting. The use of relevance information to guide the combination process is shown to reduce the variability inherent in combination of evidence.

In this paper we are mainly concerned with discussion of a formal model, based on the basic concept of divergence from information theory, for automatic query expansion. The basic principles and ideas on which our study is based are described. A theoretical framework is established, which allows the comparison and evaluation of different term scoring functions for identifying good terms for query expansion. The approaches proposed in this paper have been implemented and evaluated on collections from TREC. Preliminary results show that our approaches are viable and worthy of continued investigation.

Abstract. The application of document clustering to information retrieval has been motivated by the potential effectiveness gains postulated by the Cluster Hypothesis. The hypothesis states that relevant documents tend to be highly similar to each other, and therefore tend to appear in the same clusters. In this paper we propose that, for any given query, pairs of relevant documents will exhibit an inherent similarity which is dictated by the query itself. Our research describes an attempt to devise means by which this similarity can be detected. We propose the use of query-sensitive similarity measures that bias interdocument relationships towards pairs of documents that jointly possess attributes that are expressed in a query. We experimentally tested query-sensitive measures against conventional ones that do not take the context of the query into account. We calculated interdocument relationships for varying numbers of top-ranked documents for five document collections. Our results show a consistent and significant increase in the number of relevant documents that become nearest neighbours of any given relevant document when query-sensitive measures are used. These results suggest that the effectiveness of a cluster-based IR system has the potential to increase through the use of query-sensitive similarity measures. 1.

In this paper we report on a series of experiments designed to investigate query modification techniques motivated by the area of abductive reasoning. In particular we use the notion of abductive explanation, explanations being a description of data that highlight important features of the data. We describe several methods of creating abductive explanations, exploring term reweighting and query reformulation techniques and demonstrate their suitability for relevance feedback.

We present an analysis of word senses that provides a fresh insight into the impact of word ambiguity on retrieval effectiveness with potential broader implications for other processes of information retrieval. Using a methodology of forming artifically ambiguous words, known as pseudowords, and through reference to other researchers' work, the analysis illustrates that the distribution of the frequency of occurrance of the senses of a word plays a strong role in ambiguity's impact of effectiveness. Further investigation shows that this analysis may also be applicable to other processes of retrieval, such as Cross Language Information Retrieval, query expansion, retrieval of OCR'ed texts, and stemming. The analysis appears to provide a means of explaining, at least in part, reasons for the processes' impact (or lack of it) on effectiveness.

We analyze the kinematics of probabilistic term weights at retrieval time for different Information Retrieval models. We present four models based on different notions of probabilistic retrieval. Two of these models are based on classical probability theory and can be considered as prototypes of models long in use in Information Retrieval, like the Vector Space Model and the Probabilistic Model. The two other models are based on a logical technique of evaluating the probability of a conditional called imaging; one is a generalization of the other. We analyze the transfer of probabilities occurring in the term space at retrieval time for these four models, compare their retrieval performance using classical test collections, and discuss the results. We believe that our results provide useful suggestions on how to improve existing probabilistic models of Information Retrieval by taking into consideration term-term similarity.

article Free AccessThe potential and actual effectiveness of interactive query expansion Authors: Mark Magennis Department of Computing Science, University of Glasgow, Glasgow G12 8QQ, United Kingdom Department of Computing Science, University of Glasgow, Glasgow G12 8QQ, United KingdomView Profile , Cornelis J. van Rijsbergen Department of Computing Science, University of Glasgow, Glasgow G12 8QQ, United Kingdom Department of Computing Science, University of Glasgow, Glasgow G12 8QQ, United KingdomView Profile Authors Info & Claims ACM SIGIR ForumVolume 31Issue SIDecember 1997 pp 324–332https://doi.org/10.1145/278459.258603Published:01 July 1997Publication History 74citation957DownloadsMetricsTotal Citations74Total Downloads957Last 12 Months33Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Information is and always has been an elusive concept; nevertheless many philosophers, mathematicians, logicians and computer scientists have felt that it is fundamental. Many attempts have been made to come up with some sensible and intuitively acceptable definition of information; up to now, none of these have succeeded. This work is based on the approach followed by Dretske, Barwise, and Devlin, who claimed that the notion of information starts from the position that given an ontology of objects individuated by a cognitive agent, it makes sense to speak of the information an object (e.g., a text, an image, a video) contains about another object (e.g., the query). This phenomenon is captured by the flow of information between objects. Its exploitation is the task of an Information Retrieval system. These authors proposed a theory of information that provides an analysis of the concept of information (any type, from any media) and the manner in which intelligent organisms (referred to as cognitive agents) handle and respond to the information picked up from their environment. They defined the nature of information flow and the mechanisms that give rise to such a flow. The theory, which is based on Situation Theory, is expressed with a calculus defined on channels. The calculus was defined so that it satisfies properties that are attributed to information and its flows. This paper demonstrates the connection between this calculus and Information Retrieval, and proposes a model of an Information Retrieval system based on this calculus. © 1996 John Wiley & Sons, Inc.

In this paper we discuss the dynamics of probabilistic term

The well-known probabilistic model for information retrieval based on Bayesian conditioning of probability functions is examined. It is extended to allow conditioning based on evidence derived from the ‘passage of experience’ which may be non-propositional in nature. This latter form of conditioning is derived from Jeffrey's work on probability kinematics and it is compared with the Dempster–Shafer approach to revising belief functions whilst motivating its appropriateness for adaptive information retrieval. This new form of conditioning is combined with a non-classical logic to define a new probabilistic model for information retrieval.

'Probability is expectation founded upon partial knowledge.' (Boole, 1854)

Traditional information has relied on the extensive use of statistical parameters in the implementation of retrieval strategies. This paper sets out to investigate whether linguistic processes can be used as part of a document retrieval strategy. This is done by predefining a level of syntactic analysis of user queries only, to be used as part of the retrieval process. A large series of experiments on an experimental test collection are reported which use a parser for noun phrases as part of the retrieval strategy. The results obtained from the experiments do yield improvements in the level of retrieval effectiveness and given the crude linguistic process used and the way it was used on queries and not on document texts, suggests that the approach of using linguistic processing in retrieval, is valid.

Missing, non-applicable and imprecise values arise frequently in Office Information Systems. There is a need to treat them in a consistent and useful manner. This paper proposes a method and gives the precise semantics of the retrieval operations in a system where imprecision is allowed. It also suggests a way to handle the uncertainty introduced by imprecise data values.

A new framework based on a non-classical logic is proposed for investigating IR. The paper motivates the use of a particular conditional logic as the 'right' logic for IR. A new principle, the logical uncertainty principle, is proposed, to deal with the inherent uncertainty associated with applicable inferences.

Implicit in many information retrieval models is a logic. These logics are hardly ever formalised. This paper formalises a non-classical logic underlying information retrieval. It shows how a particular conditional logic is the ‘right’ logic to do Information Retrieval. Its relationship to existing retrieval mechanisms is investigated. The semantics of the logic are expressed in probability theory, and evaluated through a possible-world analysis, thus establishing an intensional logic. In doing so, we motivate a new principle, the logical uncertainty principle, which gives a measure of the uncertainty associated with an inference.

Underlying many of the probabilistic models for information retrieval are assumptions of stochastic dependence or independence of varying degrees of severity for the index terms describing the documents. These models generally specify a matching function, that is a function which compares a query with each document. The form of that function is to a large extent determined by the particular dependence/independence assumption. For example, if the index terms are assumed to be independently distributed over both the set of relevant and non-relevant documents then the matching function will in general be linear, whereas an assumption of dependence will lead to a non-linear function.Irrespective of the form that the matching function may take it is always assumed that the search terms in the query are known. In this paper I wish to address the problem of the choice of search terms and how this choice may be affected by an independence assumption.

No abstract available.

This paper presents the results of an experimental investigation into the effects that some forms of query expansion by term addition or term deletion, have on the retrieval effectiveness of a document retrieval system. The overall search strategy used by a user is an iterative process whereby a set of user-judged relevant documents at any point in the search is used to refine and improve on the remainder of the user's search. At some point during the search, the set of relevant documents found so far can be used to modify the original query, either by the addition or deletion of search terms. This process is called ‘query modification’ or ‘query expansion’. A number of different types of query modification strategies are tried and the results obtained are presented and analysed.

article Free Access Share on The nearest neighbour problem in information retrieval: an algorithm using upperbounds Authors: A. F. Smeaton University College Dublin, Belfield, Dublin University College Dublin, Belfield, DublinView Profile , C. J. van Rijsbergen University College Dublin, Belfield, Dublin University College Dublin, Belfield, DublinView Profile Authors Info & Claims ACM SIGIR ForumVolume 16Issue 1Summer 1981 pp 83–87https://doi.org/10.1145/1013228.511767Published:31 May 1981Publication History 45citation583DownloadsMetricsTotal Citations45Total Downloads583Last 12 Months54Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

The algorithm constructs a hierarchic classification of a set of objects. The classification is represented by a data-structure designed to facilitate searching the hierarchy. The algorithm is intended for use on growing data bases, and updates the representation as new objects or batches of objects are acquired. An application to document retrieval is outlined.