In the context of search as learning, users engage in search sessions to fill their information gaps and achieve their learning goals. Tracking the user’s state of knowledge is therefore essential for estimating how close they are to achieve these learning goals. In this respect, we extend a recently proposed approach that uses the recognition of entities present in the text to track the user’s knowledge. Our approach introduces a more complete representation by considering both the entities and their relations. More precisely, we represent both the user’s knowledge and the user’s learning goals (or target knowledge) as knowledge graphs.

A reliable representation of the user’s knowledge state during a learning search session is crucial to understand their real information needs. When a search system is aware of such a state, it can adapt the search results and provide greater support for the user’s learning objectives. A common practice to track the user’s knowledge state is to consider the content of the documents they read during their search session(s). However, most current work ignores entity mentions in the documents, which, when linked to knowledge graphs, can be a source of valuable information regarding the user’s knowledge. To fill this gap, we extend RULK—Representing User Knowledge in Search-as-Learning—with entity linking capabilities. The extended framework RULK represents and tracks user knowledge as a collection of such entities. It eventually estimates the user knowledge gain—learning outcome—by measuring the similarity between the represented knowledge and the learning objective. We show that our methods allow for up to 10% improvements when estimating user knowledge gains.

In this paper, we present a new benchmark collection that shows how 404 users’ knowledge changed over the course of a search-as-learning session. We estimate the knowledge a user gains from each visited document and monitor knowledge change on a document-by-document basis. We describe the specifics of how this collection was created and provide a use case that illustrates potential future applications.

International audience

The existing evaluation measures for information retrieval algorithms still lack awareness about the user’s cognitive aspects and their dynamics. They often consider an isolated query-document environment and ignore the user’s previous knowledge and his/her motivation behind the query. The retrieval algorithms and evaluation measures that account for those factors limit the result’s relevance to one search session, one query, or one search goal. We present a novel evaluation measure that overcomes this limitation. The framework measures the relevance of a result/document by examining its content and assessing the possible learning outcomes, for a specific user. Hence not all documents are relevant to all users. The proposed evaluation measure rewards the results’ content for their novelty with respect to what the user already knows and what has been previously proposed. The results are also rewarded for their contribution to achieving the search goals/needs. We demonstrate the efficiency of the measure by comparing it to the knowledge gain reported by 361 crowd-sourced users searching the Web across 10 different topics.