Throughout an information search, a user needs to make sense of the information found to create an understanding. This requires cognitive effort that can be demanding. Building on prior sensemaking models and expanding them with ideas from learning and cognitive psychology, we examined the use of cognitive mechanisms during individual sensemaking. We conducted a qualitative user study of 15 students who searched for and made sense of information for business analysis and news writing tasks. Through the analysis of think‐aloud protocols, recordings of screen movements, intermediate work products of sensemaking, including notes and concept maps, and final reports, we observed the use of 17 data‐driven and structure‐driven mechanisms for processing new information, examining individual concepts and relationships, and detecting anomalies. These cognitive mechanisms, as the basic operators that move sensemaking forward, provide in‐depth understanding of how people process information to produce sense. Meaningful learning and sensemaking are closely related, so our findings apply to learning as well. Our results contribute to a better understanding of the sensemaking process—how people think—and this better understanding can inform the teaching of thinking skills and the design of improved sensemaking assistants and mind tools.

Tools for automatic subject assignment help deal with scale and sustainability in creating and enriching metadata, establishing more connections across and between resources and enhancing consistency. Although some software vendors and experimental researchers claim the tools can replace manual subject indexing, hard scientific evidence of their performance in operating information environments is scarce. A major reason for this is that research is usually conducted in laboratory conditions, excluding the complexities of real‐life systems and situations. The article reviews and discusses issues with existing evaluation approaches such as problems of aboutness and relevance assessments, implying the need to use more than a single “gold standard” method when evaluating indexing and retrieval, and proposes a comprehensive evaluation framework. The framework is informed by a systematic review of the literature on evaluation approaches: evaluating indexing quality directly through assessment by an evaluator or through comparison with a gold standard, evaluating the quality of computer‐assisted indexing directly in the context of an indexing workflow, and evaluating indexing quality indirectly through analyzing retrieval performance.

This study demonstrates an improved conceptual foundation to support well‐structured analysis of image topicality. First we present a conceptual framework for analyzing image topicality, explicating the layers, the perspectives, and the topical relevance relationships involved in modeling the topicality of art images. We adapt a generic relevance typology to image analysis by extending it with definitions and relationships specific to the visual art domain and integrating it with schemes of image‐text relationships that are important for image subject indexing. We then apply the adapted typology to analyze the topical relevance relationships between 11 art images and 768 image tags assigned by art historians and librarians. The original contribution of our work is the topical structure analysis of image tags that allows the viewer to more easily grasp the content, context, and meaning of an image and quickly tune into aspects of interest; it could also guide both the indexer and the searcher to specify image tags/descriptors in a more systematic and precise manner and thus improve the match between the two parties. An additional contribution is systematically examining and integrating the variety of image‐text relationships from a relevance perspective. The paper concludes with implications for relational indexing and social tagging.

This review introduces a comprehensive model of the cognitive process and mechanisms of individual sensemaking to provide a theoretical basis for: empirical studies that improve our understanding of the cognitive process and mechanisms of sensemaking and integration of results of such studies; education in critical thinking and sensemaking skills; the design of sensemaking assistant tools that support and guide users. The paper reviews and extends existing sensemaking models with ideas from learning and cognition. It reviews literature on sensemaking models in human‐computer interaction ( HCI ), cognitive system engineering, organizational communication, and library and information sciences ( LIS ), learning theories, cognitive psychology, and task‐based information seeking. The model resulting from this synthesis moves to a stronger basis for explaining sensemaking behaviors and conceptual changes. The model illustrates the iterative processes of sensemaking, extends existing models that focus on activities by integrating cognitive mechanisms and the creation of instantiated structure elements of knowledge, and different types of conceptual change to show a complete picture of the cognitive processes of sensemaking. The processes and cognitive mechanisms identified provide better foundations for knowledge creation, organization, and sharing practices and a stronger basis for design of sensemaking assistant systems and tools.

Synthesizing and building on many ideas from the literature, this article presents an improved conceptual framework that clarifies the notion of relevance with its many elements, variables, criteria, and situational factors. Relevance is defined as a Relationship (R) between an Information Object (I) and an Information Need (N) (which consists of Topic, User, Problem/Task, and Situation/Context) with focus on R. This defines Relevance‐as‐is (conceptual relevance, strong relevance). To determine relevance, an Agent A (a person or system) operates on a representation I′ of the information object and a representation N′ of the information need, resulting in relevance‐as‐determined (operational measure of relevance, weak relevance, an approximation). Retrieval tests compare relevance‐as‐determined by different agents. This article discusses and compares two major approaches to conceptualizing relevance: the entity‐focused approach (focus on elaborating the entities involved in relevance) and the relationship‐focused approach (focus on explicating the relational nature of relevance). The article argues that because relevance is fundamentally a relational construct the relationship‐focused approach deserves a higher priority and more attention than it has received. The article further elaborates on the elements of the framework with a focus on clarifying several critical issues on the discourse on relevance.

Abstract The authors describe user interface tools based on search histories to support legal information seekers. The design of the tools was informed by the results of a user study (Komlodi, 2002a) that examined the use of human memory, external memory aids, and search histories in legal information seeking and derived interface design recommendations for information storage and retrieval systems. The data collected were analyzed to identify potential task areas where search histories can support information seeking and use. The results show that many information‐seeking tasks can take advantage of automatically and manually recorded history information. These findings encouraged the design of user interface tools building on search history information: direct search history displays, history‐enabled scratchpad facilities, and organized results collection tools.

Test collections model use cases in ways that facilitate evaluation of information retrieval systems. This paper describes the use of search-guided relevance assessment to create a test collection for retrieval of spontaneous conversational speech. Approximately 10,000 thematically coherent segments were manually identified in 625 hours of oral history interviews with 246 individuals. Automatic speech recognition results, manually prepared summaries, controlled vocabulary indexing, and name authority control are available for every segment. Those features were leveraged by a team of four relevance assessors to identify topically relevant segments for 28 topics developed from actual user requests. Search-guided assessment yielded sufficient inter-annotator agreement to support formative evaluation during system development. Baseline results for ranked retrieval are presented to illustrate use of the collection.

Much is known about the design of automated systems to search broadcast news, but it has only recently become possible to apply similar techniques to large collections of spontaneous speech. This paper presents initial results from experiments with speech recognition, topic segmentation, topic categorization, and named entity detection using a large collection of recorded oral histories. The work leverages a massive manual annotation effort on 10 000 h of spontaneous speech to evaluate the degree to which automatic speech recognition (ASR)-based segmentation and categorization techniques can be adapted to approximate decisions made by human annotators. ASR word error rates near 40% were achieved for both English and Czech for heavily accented, emotional and elderly spontaneous speech based on 65-84 h of transcribed speech. Topical segmentation based on shifts in the recognized English vocabulary resulted in 80% agreement with manually annotated boundary positions at a 0.35 false alarm rate. Categorization was considerably more challenging, with a nearest-neighbor technique yielding F=0.3. This is less than half the value obtained by the same technique on a standard newswire categorization benchmark, but replication on human-transcribed interviews showed that ASR errors explain little of that difference. The paper concludes with a description of how these capabilities could be used together to search large collections of recorded oral histories.

Classifications/ontologies, thesauri, and dictionaries serve many functions, which are summarized in this note. As a result of this multiplicity of functions, classifications—often called ontologies—are developed in many communities of research and practice. Unfortunately, there is little communication and mutual learning; thus, efforts are fragmented, resulting in considerable reinvention and less than optimal products.

This article proposes a model of document selection by real users of a bibliographic retrieval system. It reports on Part 1 of a longitudinal study of decision making on document use by academics during an actual research project. (Part 2 followed up the same users on how the selected documents were actually used in subsequent stages.) The participants are 25 self-selected faculty and graduate students in Agricultural Economics. After a reference interview, the researcher conducted a search of DIALOG databases and prepared a printout. The users selected documents from this printout; they were asked to read and think aloud while selecting documents. Their verbal reports were recorded and analyzed from a utility-theoretic perspective. The following model of the decision-making in the selection process emerged: document information elements (DIEs) in document records provide the information for judging documents on 11 criteria (including topicality, orientation, quality, novelty, and authority); the criteria judgments are combined in an assessment of document value along five dimensions (epistemic, functional, conditional, social, and emotional values), leading to the use decision. This model accounts for the use of personal knowledge and decision strategies applied in the selection process. The model has implications for the design of an intelligent document selection assistant. © 1998 John Wiley & Sons, Inc.

This article presents a logical analysis of the characteristics of indexing and their effects on retrieval performance. It establishes the ability to ask the questions one needs to ask as the foundation of performance evaluation, and recall and discrimination as the basic quantitative performance measures for binary noninteractive retrieval systems. It then defines the characteristics of indexing that affect retrieval—namely, indexing devices, viewpoint-based and importance-based indexing exhaustivity, indexing specificity, indexing correctness, and indexing consistency—and examines in detail their effects on retrieval. It concludes that retrieval performance depends chiefly on the match between indexing and the requirements of the individual query and on the adaptation of the query formulation to the characteristics of the retrieval system, and that the ensuing complexity must be considered in the design and testing of retrieval systems. © 1994 John Wiley & Sons, Inc.

prototype system that demonstrates this potential is described.

Abstract This article presents a conceptual framework for the organization of factors (independent variables) affecting online bibliographic retrieval; the variables were collected from major sources. The first part describes the various roles that variables play in a research study. The second part gives the conceptual framework for the factors with examples of individual variables for illustration. We consider the following elements of the total retrieval situation: the setting, the user, the request, the database, the search system, the searcher, the search process, and the search outcome. For each of these elements (excluding search outcome) a detailed list of variables is given in the Appendix. The variables are organized in a table according to themes that are applicable across elements.

Abstract Scope: The purpose of this study was to explore the use and effectiveness of current awareness (CA) methods, especially SDI, among chemists at Canadian universities. Method: We used a questionnaire which was mailed to a sample of 170 (response rate 80%). The variables studied include perceived scatter of information, use of various CA methods, time spent on CA, perceived success in CA, and CA efficiency = success/time spent. The results of this explanatory study are tentative. Major results: Keeping up‐to‐date is more difficult in specialties with high scatter of information (hypothesis 5). SDI seems to be the only method capable of counteracting scatter (hypotheses 5–7). In contrast, if scatter is low, SDI use decreases CA efficiency (hypothesis 6a); scanning journals, the most prevalent CA method, seems to be sufficient for low‐scatter specialties. High scatter does not lead to increased SDI use, in some cases due to perceived difficulty of profile construction (hypothesis 3a). Conclusions: SDI services should concentrate efforts on serving scientists in specialties with high scatter of information by developing systems that are capable of searching across conventional disciplines and specialties, by providing assistance in profile development, and by conducting a marketing campaign directed toward these scientists.

Abstract This paper is in response to William S. Cooper: “On Selecting a Measure of Retrieval Effectiveness.” Journal of the American Society for Information Science. 1973;24(2)87–100. Whereas Cooper considers (subjective) satisfaction of a user as the utility of an individual search (and then computes system utility as the average search utility), this paper argues that improvement in the task performance of the user is a much more appropriate measure of utility. From this it is shown that recall, while unimportant or even harmful in many search situations, is of vital importance in others. This is in contrast to Cooper's view that recall by itself is not a meaningful measure of system performance at all.

Abstract The construction of a Universal Source Thesaurus (UST) is suggested. Unlike a universal classification, UST's main purpose would not be to serve as an indexing language to be applied directly on an universal scale but to: serve as a source in the construction of special schemes; serve as a concordance between a great many classification schemes, existing ones as well as new ones. The following problems arising in the planning for UST are dealt with briefly: The structure of UST–UST would have to be as complete and detailed as possible. The use of UST in the construction of special schemes—especially the “extraction” of special schemes. The use of UST as a concordance or “switching language” between different classification schemes as a basis for cooperative cataloguing and the possibility to have special schemes common to a group of related institutions (semi‐universal classifications). Organization and management of UST.