Traditional search result evaluation metrics in information retrieval, such as MAP and NDCG, naively focus on topical relevance between a document and search topic and assume this relationship as mono-dimensional and often binary. They neglect document content overlap and assume gains piling up as the searcher examines the ranked list at greater length. We propose a novel search result evaluation framework based on multidimensional, graded relevance assessments, explicit modelling of document overlaps and attributes affecting document usability beyond relevance. Document relevance to a search task is seen to consist of several content themes and document usability attributes. Documents may also overlap regarding their content themes. Attributes such as document readability, trustworthiness, or language represent the entire document’s usability in the search task context, for a given searcher and her motivating task. The proposed framework evaluates the quality of a ranked search result, taking into account the contribution of each successive document, with estimated overlap across themes, and usability based on its attributes.

Abstract The aim of the study was to develop an empirical measure for everyday information mastering (EIM). EIM describes the ways that individuals, based on their beliefs, attitudes, and expectations, orient themselves to information as a resource of everyday action. The key features of EIM were identified by conceptual analysis focusing on three EIM frameworks. Four modes of EIM—Proactive, Social, Reactive, and Passive—and their 12 constituents were identified. A survey of 39 items was developed in two pilot studies to operationalize the identified modes as measurable EIM constituents. The respondents in the main study were upper secondary school students (n = 412). Exploratory factor analysis (EFA) was applied to validate subscales for each EIM constituent. Seven subscales emerged: Inquiring and Scanning in the Proactive mode, Social media‐centered, and Experiential in the Social mode, and Information poor, Overwhelmed, and Blunting in the Passive mode. Two constituents, Serendipitous and Intuitive, were not supported in the EFA. The findings highlight that the core constituents of an individual's everyday information mastering can be operationalized as psychometric scales. The instrument contributes to the systematic empirical study of EIM constituents and their relationships. The study further sheds light on key modes of EIM.

The aim of the study was to test whether query expansion by approximate string matching methods is beneficial in retrieval from historical newspaper collections in a language rich with compounds and inflectional forms (Finnish). First, approximate string matching methods were used to generate lists of index words most similar to contemporary query terms in a digitized newspaper collection from the 1800s. Top index word variants were categorized to estimate the appropriate query expansion ranges in the retrieval test. Second, the effectiveness of approximate string matching methods, automatically generated inflectional forms, and their combinations were measured in a Cranfield‐style test. Finally, a detailed topic‐level analysis of test results was conducted. In the index of historical newspaper collection the occurrences of a word typically spread to many linguistic and historical variants along with optical character recognition (OCR) errors. All query expansion methods improved the baseline results. Extensive expansion of around 30 variants for each query word was required to achieve the highest performance improvement. Query expansion based on approximate string matching was superior to using the inflectional forms of the query words, showing that coverage of the different types of variation is more important than precision in handling one type of variation.

Evaluation is central in research and development of information retrieval (IR). In addition to designing and implementing new retrieval mechanisms, one must also show through rigorous evaluation that they are effective. A major focus in IR is IR mechanisms’ capability of ranking relevant documents optimally for the users, given a query. Searching for information in practice involves searchers, however, and is highly interactive. When human searchers have been incorporated in evaluation studies, the results have often suggested that better ranking does not necessarily lead to better search task, or work task, performance. Therefore, it is not clear which system or interface features should be developed to improve the effectiveness of human task performance. In the present article, we focus on the evaluation of task-based information interaction (TBII). We give special emphasis to learning tasks to discuss TBII in more concrete terms. Information interaction is here understood as behavioral and cognitive activities related to task planning, searching information items, selecting between them, working with them, and synthesizing and reporting. These five generic activities contribute to task performance and outcome and can be supported by information systems. In an attempt toward task-based evaluation, we introduce program theory as the evaluation framework. Such evaluation can investigate whether a program consisting of TBII activities and tools works and how it works and, further, provides a causal description of program (in)effectiveness. Our goal in the present article is to structure TBII on the basis of the five generic activities and consider the evaluation of each activity using the program theory framework. Finally, we combine these activity-based program theories in an overall evaluation framework for TBII. Such an evaluation is complex due to the large number of factors affecting information interaction. Instead of presenting tested program theories, we illustrate how the evaluation of TBII should be accomplished using the program theory framework in the evaluation of systems and behaviors, and their interactions, comprehensively in context.

Source‐based writing assignments conducted by groups of students are a common learning task used in information literacy instruction. The fundamental assumption in group assignments is that students' collaboration substantially enhances their learning. The present study focused on the group work strategies adopted by upper secondary school students in source‐based writing assignments. Seventeen groups authored Wikipedia or Wikipedia‐style articles and were interviewed during and after the assignment. Group work strategies were analyzed in 6 activities: planning, searching, assessing sources, reading, writing, and editing. The students used 2 cooperative strategies: delegation and division of work, and 2 collaborative strategies: pair and group collaboration. Division of work into independently conducted parts was the most popular group work strategy. Also group collaboration, where students worked together to complete an activity, was commonly applied. Division of work was justified by efficiency in completing the project and by ease of control in the fair division of contributions. The motivation behind collaboration was related to quality issues and shared responsibility. We suggest that the present designs of learning tasks lead students to avoid collaboration, increasing the risk of low learning outcomes in information literacy instruction.

Abstract In this paper, we focus on the effect of graded relevance on the results of interactive information retrieval (IR) experiments based on assigned search tasks in a test collection. A group of 26 subjects searched for four Text REtrieval Conference (TREC) topics using automatic and interactive query expansion based on relevance feedback. The TREC‐ and user‐suggested pools of relevant documents were reassessed on a four‐level relevance scale. The results show that the users could identify nearly all highly relevant documents and about half of the marginal ones. Users also selected a fair number of irrelevant documents for query expansion. The findings suggest that the effectiveness of query expansion is closely related to the searchers' success in retrieving and identifying highly relevant documents for feedback. The implications of the results on interpreting the findings of past experiments with liberal relevance thresholds are also discussed.

Most test collections (like TREC and CLEF) for experimental research in information retrieval apply binary relevance assessments. This paper introduces a four-point relevance scale and reports the findings of a project in which TREC-7 and TREC-8 document pools on 38 topics were reassessed. The goal of the reassessment was to build a subcollection of TREC for experiments on highly relevant documents and to learn about the assessment process as well as the characteristics of a multigraded relevance corpus.Relevance criteria were defined so that a distinction was made between documents rich in topical information (relevant and highly relevant documents) and poor in topical information (marginally relevant documents). It turned out that about 50% of documents assessed as relevant were regarded as marginal. The characteristics of the relevance corpus and lessons learned from the reassessment project are discussed. The need to develop more elaborated relevance assessment schemes is emphasized.

The Interactive Query Performance Analyser (QPA) for information retrieval systems is a Web-based tool for analysing and comparing the performance of individual queries. On top of a standard test collection, it gives an instant visualisation of the performance achieved in a given search topic by any user-generated query. In addition to experimental IR research, QPA can be used in user training to demonstrate the characteristics of and compare differences between IR systems and searching strategies. The first prototype (versions 3.0 and 3.5) of the Query Performance Analyser was developed at the Department of Information Studies, University of Tampere, to serve as a tool for rapid query performance analysis, comparison and visualisation [4,5]. Later, it has been applied to interactive optimisation of queries [2,3]. The analyser has served also in learning environments for IR [1].The demonstration is based on the newest version of the Query Performance Analyser (v. 5.1). It is interfaced to a traditional Boolean IR system (TRIP) and a probabilistic IR system (Inquery) providing access to the TREC collection and two Finnish test collections. Version 5.1 supports multigraded relevance scales, new types of performance visualisations, and query conversions based on mono- and multi-lingual dictionaries. The motivation in developing the analyser is to emphasise the necessity of analysing the behaviour of individual queries. Information retrieval experiments usually measure the average effectiveness of IR methods developed. The analysis of individual queries is neglected although test results may contain individual test topics where general findings do not hold. For the real user of an IR system, the study of variation in results is even more important than averages.

Traditional methods for the system-oriented evaluation of Boolean IR system suffer from validity and reliability problems. Laboratory-based research neglects the searcher and studies suboptimal queries. Research on operational systems fails to make a distinction between searcher performance and system performance. This approach is neither capable of measuring performance at standard points of operation (e.g. across R0.0-R1.0).

We present a deductive data model for conceptbased query expansion.It ]s based on three abstraction levels: the conceptual, linguistic and occurrence levels.Concepts and relationships among them are represented at the conceptual level.The expression level represents natural language expressions for concepts.Each expression has one or more matchmg models at the occurrence level.The models specify the matching of the expression in database Indices built in varying ways.The data model supports a concept-based query expansion and formulation tool, the ExpansionTool, for heterogeneous IR system envirom ments, Expansion is controlled by adjustable matching reliability.