Information relevance judgment is a fundamental cognitive decision performed by users during information seeking and reading tasks.Inferring relevance decisions from neurophysiological signals, particularly through eye-tracking and EEG recordings, opens possibilities for real-time relevance detection.To support this line of research, we present g-Rel-READER, a unique dataset comprising eye-tracking and EEG recordings collected during reading and relevance judgment of short news stories in a question-answering task.The dataset provides screen coordinates for each word, enabling temporal and spatial alignment between eye-tracking data, EEG signals, and the specific words being read.Question-relevant words are explicitly marked, facilitating analysis of relevance judgment processes.

Assessment of text relevance is an important aspect of human–information interaction. For many search sessions it is essential to achieving the task goal. This work investigates text relevance decision dynamics in a question‐answering task by direct measurement of eye movement using eye‐tracking and brain activity using electroencephalography EEG. The EEG measurements are correlated with the user's goal‐directed attention allocation revealed by their eye movements. In a within‐subject lab experiment ( N = 24), participants read short news stories of varied relevance. Eye movement and EEG features were calculated in three epochs of reading each news story (early, middle, final) and for periods where relevant words were read. Perceived relevance classification models were learned for each epoch. The results show reading epochs where relevant words were processed could be distinguished from other epochs. The classification models show increasing divergence in processing relevant vs. irrelevant documents after the initial epoch. This suggests differences in cognitive processes used to assess texts of varied relevance levels and provides evidence for the potential to detect these differences in information search sessions using eye tracking and EEG.

User domain knowledge affects search behaviors and search success. Predicting a user's knowledge level from implicit evidence such as search behaviors could allow an adaptive information retrieval system to better personalize its interaction with users. This study examines whether user domain knowledge can be predicted from search behaviors by applying a regression modeling analysis method. We identify behavioral features that contribute most to a successful prediction model. A user experiment was conducted with 40 participants searching on task topics in the domain of genomics. Participant domain knowledge level was assessed based on the users' familiarity with and expertise in the search topics and their knowledge of MeSH (Medical Subject Headings) terms in the categories that corresponded to the search topics. Users' search behaviors were captured by logging software, which includes querying behaviors, document selection behaviors, and general task interaction behaviors. Multiple regression analysis was run on the behavioral data using different variable selection methods. Four successful predictive models were identified, each involving a slightly different set of behavioral variables. The models were compared for the best on model fit, significance of the model, and contributions of individual predictors in each model. Each model was validated using the split sampling method. The final model highlights three behavioral variables as domain knowledge level predictors: the number of documents saved, the average query length, and the average ranking position of the documents opened. The results are discussed, study limitations are addressed, and future research directions are suggested.

Personalization of support for information seeking depends crucially on the information retrieval system's knowledge of the task that led the person to engage in information seeking. Users work during information search sessions to satisfy their task goals, and their activity is not random. To what degree are there patterns in the user activity during information search sessions? Do activity patterns reflect the user's situation as the user moves through the search task under the influence of his or her task goal? Do these patterns reflect aspects of different types of information-seeking tasks? Could such activity patterns identify contexts within which information seeking takes place? To investigate these questions, we model sequences of user behaviors in two independent user studies of information search sessions (N = 32 users, 128 sessions, and N = 40 users, 160 sessions). Two representations of user activity patterns are used. One is based on the sequences of page use; the other is based on a cognitive representation of information acquisition derived from eye movement patterns in service of the reading process. One of the user studies considered journalism work tasks; the other concerned background research in genomics using search tasks taken from the TREC Genomics Track. The search tasks differed in basic dimensions of complexity, specificity, and the type of information product (intellectual or factual) needed to achieve the overall task goal. The results show that similar patterns of user activity are observed at both the cognitive and page use levels. The activity patterns at both representation layers are able to distinguish between task types in similar ways and, to some degree, between tasks of different levels of difficulty. We explore relationships between the results and task difficulty and discuss the use of activity patterns to explore events within a search session. User activity patterns can be at least partially observed in server-side search logs. A focus on patterns of user activity sequences may contribute to the development of information systems that better personalize the user's search experience.

Can the activity patterns of page use during information search sessions discriminate between different types of information seeking tasks? We model sequences of interactions with search result and content pages during information search sessions. Two representations are created: the sequences of page use and a cognitive representation of page interactions. The cognitive representation is based on logged eye movement patterns of textual information acquisition via the reading process. Page sequence actions from task sessions (n=109) in a user study are analyzed. The study tasks differed from one another in basic dimensions of complexity, specificity,level, and the type of information product (intellectual or factual). The results show that differences in task types can be measured at both the level of observations of page type sequences and at the level of cognitive activity on the pages. We discuss the implications for personalization of search systems, measurement of task similarity and the development of user-centered information systems that can support the user's current and expected search intentions.

We report on an investigation of behavioral differences between users in difficult and easy search tasks. Behavioral factors that can be used in real-time to predict task difficulty are identified. User data was collected in a controlled lab experiment (n=38) where each participant completed four search tasks in the genomics domain. We looked at user behaviors that can be obtained by systems at three levels, distinguished by the time point when the measurements can be done. They are: 1) first-round level at the beginning of the search, 2) accumulated level during the search, and 3) whole-session level by the end of the search. Results show that a number of user behaviors at all three levels differed between easy and difficult tasks. Models predicting task difficulty at all three levels were developed and evaluated. A real-time model incorporating first-round and accumulated levels of behaviors (FA) had fairly good prediction performance (accuracy 83%; precision 88%), which is comparable with the model using the whole-session level behaviors which are not real-time (accuracy 75%; precision 92%). We also found that for efficiency purpose, using only a limited number of significant variables (FC_FA) can obtain a prediction accuracy of 75%, with a precision of 88%. Our findings can help search systems predict task difficulty and adapt search results to users.

Personalization of search results offers the potential for significant improvement in information retrieval performance. User interactions with the system and documents during information-seeking sessions provide a wealth of information about user preferences and their task goals. In this paper, we propose methods for analyzing and modeling user search behavior in search sessions to predict document usefulness and then using information to personalize search results. We generate prediction models of document usefulness from behavior data collected in a controlled lab experiment with 32 participants, each completing uncontrolled searching for 4 tasks in the Web. The generated models are then tested with another data set of user search sessions in radically different search tasks and constrains. The documents predicted useful and not useful by the models are used to modify the queries in each search session using a standard relevance feedback technique. The results show that application of the models led to consistently improved performance over a baseline that did not take account of user interaction information. These findings have implications for designing systems for personalized search and improving user search experience.

Click through events in search results pages (SERPs) are not reliable implicit indicators of document relevance. A user's task and domain knowledge are key factors in recognition and link selection and the most useful SERP document links may be those that best match the user's domain knowledge. User study participants rated their knowledge of genomics MeSH terms before conducting 2004 TREC Genomics Track tasks. Each participant's document knowledge was represented by their knowledge of the indexing MeSH terms. Results show high, intermediate, and low domain knowledge groups had similar document selection SERP rank distributions. SERP link selection distribution varied when participant knowledge of the available documents was analyzed. High domain knowledge participants usually selected a document with the highest personal knowledge rating. Low domain knowledge participants were reasonably successful at selecting available documents of which they had the most knowledge, while intermediate knowledge participants often failed to do so. This evidence for knowledge effects on SERP link selection may contribute to understanding the potential for personalization of search results ranking based on user domain knowledge.

This study uses regression modeling to predict a user's domain knowledge level (DK) from implicit evidence provided by certain search behaviors. A user study (n=35) with recall-oriented search tasks in the genomic domain was conducted. A number of regression models of a person's DK, were generated using different behavior variable selection methods. The best model highlights three behavior variables as DK predictors: the number of documents saved, the average query length, and the average ranking position of documents opened. The model is validated using the split sampling method. Limitations and future research directions are discussed.

We report on an investigation into people’s behaviors on information search tasks, specifically the relation between eye movement patterns and task characteristics. We conducted two independent user studies (n = 32 and n = 40), one with journalism tasks and the other with genomics tasks. The tasks were constructed to represent information needs of these two different users groups and to vary in several dimensions according to a task classification scheme. For each participant we classified eye gaze data to construct models of their reading patterns. The reading models were analyzed with respect to the effect of task types and Web page types on reading eye movement patterns. We report on relationships between tasks and individual reading behaviors at the task and page level. Specifically we show that transitions between scanning and reading behavior in eye movement patterns and the amount of text processed may be an implicit indicator of the current task type facets. This may be useful in building user and task models that can be useful in personalization of information systems and so address design demands driven by increasingly complex user actions with information systems. One of the contributions of this research is a new methodology to model information search behavior and investigate information acquisition and cognitive processing in interactive information tasks