Online questionnaires are efficient tools to collect self-report data. Yet, their quality depends on how comprehensible the items are. Eye tracking provides an objective way to assess item comprehensibility. In this paper, we present initial research on the automated detection of comprehension problems in questionnaire items. In a study with 49 participants, we recorded eye movements while they read 32 items in affirmative and negated versions. We defined two prediction tasks: (1) detecting comprehension difficulties for individual respondents, and (2) identifying problematic items based on aggregated predictions. Across eight models and four feature sets, our best-performing classifiers, SGD and Ridge Classifier, outperformed single-feature baselines. Furthermore, an automatically-selected feature set achieved performance comparable to the best-performing feature set using all features and surpassed the literature-inspired feature set. Our findings provide initial evidence that machine learning models trained on eye-tracking data can support automated detection of comprehension problems in questionnaire items.

Abstract Searching for information on the web has become an essential task in our society. This article presents a preregistered experimental study that investigates how the knowledge dimension (conceptual, procedural) and relevance of displaying spatiotemporal changes (high, low) for a given search task affect searchers' preference for different resource modalities (websites, videos). Additionally, effects of source reputation (high, low) were examined. Participants were confronted with 16 learning tasks and respective mock‐up video or website search results and had to indicate the likelihood of selecting the respective search results. The learning tasks varied regarding their knowledge dimension and their degree of spatiotemporal changes. Search results varied regarding source reputation and resource modality. Study results showed that search results with low source reputation were more likely to be selected when they were video results as compared to website results. Furthermore, for learning tasks with a high degree of spatiotemporal changes, a preference for videos over website results was found, while a low degree of spatiotemporal changes did not lead to modality preferences. To conclude, both knowledge dimension and degree of visuospatial changes of learning tasks seem to be promising classifications to consider for understanding users' source and modality selection during web‐based learning.

Today’s search engines allow users to discover relevant information in different types of modalities or media, e.g., web pages, text documents, images, or videos. It is, however, a challenging task to present mixed-modality result lists in an effective and easy-to-skim form. The two most commonly used approaches are to present the modalities side-by-side, each in a separate column of the result page; or to separate the modalities into multiple tabs. However, the field lacks a structured investigation on how the column or tab layout influence the users’ perception and usage of multimodal resources in an academic search task. In this paper, we present a user study (N=50) where the participants were asked to accomplish a search task for a fictive computer science seminar at the university. We evaluate the influence of the different layouts on (1) user search behavior (e.g., time until first resource is saved) and (2) the relevance of the selected resources for the task at hand. Finally, we discuss the results and possible implications for the design of multimodal search result presentation.

This study aimed to investigate the roles of a text-highlighting tool and readers’ re-reading behaviour in their integrated understanding of multiple documents. University students (N = 95) read five partly conflicting documents on a health-related issue on a touch display with or without a text-highlighting tool. Integrated understanding of documents was assessed by the number of intertextual connections in essays written after reading and by a source-content mapping task. The provision of the text-highlighting tool resulted in longer initial reading times even when subtracting the time taken for highlighting, but shorter re-reading times, particularly for participants with a high number of re-readings. Further, only for participants with a high number of re-readings, the provision of the text-highlighting tool resulted in more intertextual connections than when no text-highlighting tool was provided. Participants’ source-content integration was positively related to the number of re-readings, regardless of whether the text-highlighting tool was provided. Finally, additional exploratory eye-tracking analyses revealed that for two out of the five documents, participants in the with-highlighting condition focused on significantly smaller parts of the documents during re-reading than controls.

The present study examined reading interactions and multiple document comprehension of 108 university students who read five partly conflicting documents on a health-related issue. Documents were presented on a large touch interface that either enabled a simultaneous or imposed a sequential presentation of the documents. None of the three multiple document comprehension measures (number of intertextual connections in essays, score in source-content mapping task, and number of source names recalled) was affected by document presentation. However, in the sequential condition, the number of revisits to documents was positively related to memory for source names and source-content integration, but not to intertextual integration. Conversely, in the simultaneous condition, participants who grouped documents during reading showed greater intertextual and source-content integration, but no greater memory for source names than those who had not grouped documents. To conclude, reading interactions played a more important role in readers’ multiple document comprehension than document presentation.

Supervised machine learning utilizes large datasets, often with ground truth labels annotated by humans. While some data points are easy to classify, others are hard to classify, which reduces the inter-annotator agreement. This causes noise for the classifier and might affect the user’s perception of the classifier’s performance. In our research, we investigated whether the classification difficulty of a data point influences how strongly a prediction mistake reduces the “perceived accuracy”. In an experimental online study, 225 participants interacted with three fictive classifiers with equal accuracy (73%). The classifiers made prediction mistakes on three different types of data points (easy, difficult, impossible). After the interaction, participants judged the classifier’s accuracy. We found that not all prediction mistakes reduced the perceived accuracy equally. Furthermore, the perceived accuracy differed significantly from the calculated accuracy. To conclude, accuracy and related measures seem unsuitable to represent how users perceive the performance of classifiers.

The emerging research field Search as Learning (SAL) investigates how the Web facilitates learning through modern information retrieval systems. SAL research requires significant amounts of data that capture both search behavior of users and their acquired knowledge in order to obtain conclusive insights or train supervised machine learning models. However, the creation of such datasets is costly and requires interdisciplinary efforts in order to design studies and capture a wide range of features. In this paper, we address this issue and introduce an extensive dataset based on a user study, in which 114 participants were asked to learn about the formation of lightning and thunder. Participants’ knowledge states were measured before and after Web search through multiple-choice questionnaires and essay-based free recall tasks. To enable future research in SAL-related tasks we recorded a plethora of features and person-related attributes. Besides the screen recordings, visited Web pages, and detailed browsing histories, a large number of behavioral features and resource features were monitored. We underline the usefulness of the dataset by describing three, already published, use cases.

This pilot study analyzes the reading patterns of 15 German students while receiving instant messages through a smartphone, imitating an online conversation. With this pilot study, we aim to test the eye-tracking setup and methodology employed, in which we analyze specifically the moment in which participants return to the reading after answering the instant messages. We explore the relationships with reading comprehension performance and differences across readers, considering individual differences regarding reading habits and multitasking behavior.

Eye-tracking research is beneficial for better understanding user behaviour in search engines. The present paper presents a comprehensive narrative literature review of eye-tracking studies examining factors influencing users’ viewing behaviour on results pages of search engines. Discipline-specific databases from Psychology, Computer Science, and Library and Information Science, as well as one multidisciplinary database have been searched for relevant articles. Criteria for inclusion were that a paper reported empirical results from an eye-tracking study in which effects of a specific factor on users’ viewing behaviour on search engine results pages (SERPs) were examined, with inferential statistical results being reported. This led to a set of 41 papers that were further examined. The papers were grouped into three categories according to three types of factors that may affect individuals’ web search activities: contextual factors, resource factors, and individual factors. Papers were assigned to these categories and subsequently to sub-categories. Overall, while for some sub-categories robust findings can be reported, we found results in many sub-categories to be inconclusive. For future research, we recommend a shift from small-scale studies examining single factors to more comprehensive and theory-driven research using larger sample sizes.

In this study we examined the influence of device type (i.e., laptop vs. smartphone) on N = 49 secondary-school students' search query length and number of queries used in a Searching as Learning (SAL) task addressing a controversial socio-scientific issue (i.e., the use of nuclear power to solve the climate crisis). Furthermore, we examined the influence of device type on students' learning outcomes by analyzing the number of pro and con arguments included in a subsequent essay task. Results showed that on laptops students used more queries than on smartphones. In contrast, no differences were found in search query length or in the number of pro and con arguments in the essay. Moreover, irrespective of device type, average search query length was positively associated with the number of con arguments. To conclude, the present study provides first indications that smartphone usage could be incorporated in the classroom for Web inquiry tasks.

The present paper reports on a Searching as Learning (SAL) study, with 115 university students learning about the formation of thunderstorms and lightning by searching the web. Participants' knowledge on the topic was measured before (t1) and after the learning phase (t2) with a free essay task. During the web search, the time participants spent on different information resources was measured, differentiating between text-dominated websites and videos. Furthermore, to examine the role of learners' cognitive abilities, their working memory capacity (WMC) and reading comprehension ability (RCA) were assessed. Results of a hierarchical linear regression analysis showed that after controlling for prior knowledge, WMC and RCA positively predicted students' learning outcome, as measured by the number of correct concepts included in essays composed after search. Moreover, the time spent on text-dominated websites was also associated with better learning outcomes. In contrast, the time spent watching videos was not beneficial. To conclude, our findings provide preliminary evidence that the time leaners spend on text-dominated websites and on videos might not contribute equally to the acquisition of conceptual knowledge in SAL tasks.

Web search is one of the most ubiquitous online activities and often used for learning purposes, i.e., to extend one's knowledge or skills about certain topics or procedures. The importance of learning as an outcome of Web search has been recognized in research at the intersection of information retrieval, human-computer interaction, psychology, and educational sciences. Search as Learning (SAL) research examines relationships between querying, navigation, and reading behavior during Web search and the resulting learning outcomes, and how they can be measured, predicted, and supported. IWILDS aims to provide a platform to the interdisciplinary SAL community, with the objective to bring together interested researchers, provide room for presentation and discussion of novel research insights, and to inspire future directions of SAL research.

Experimental research investigating the processes or influencing factors of diagnostic reasoning or diagnostic success is predominantly conducted with case descriptions spanning no more than one A4 page. In this paper, we argue for a more authentic task setting in the form of multiple documents case descriptions, and make suggestions how to design them to suit different research questions. We further review methods used in previous studies, such as think-aloud protocols and written justifications of diagnoses, and discuss how they can be used in order to assess the cognitive processes underlying diagnostic reasoning in more detail. Additionally, based on findings from the field of multiple documents comprehension, we outline how participants' gaze behavior on and their interaction with the documents might also be used to assess processes of information comparison and corroboration during reading as part of participants' diagnostic reasoning process.

Eye-tracking technology was used to examine Internet search result evaluation strategies adopted by sixth-grade students (N = 36) during ten experimental information search tasks. The relevancy of the search result’s title, URL, and snippet components was manipulated and selection of search results as well as looking into probabilities on the search result components was analysed. The results revealed that during first-pass inspection, students read the search engine page by first looking at the title of a search result. If the title was relevant, the probability of looking at the snippet of the search result increased. During second-pass inspection, there was a high probability of students focusing on the most promising search result by inspecting all of its components before making their selection. A cluster analysis revealed three viewing strategies: half of the students looked mainly at the titles and snippets; one-third with high probability examined all components; and one-sixth mainly focused on titles, leading to more frequent errors in search result selection. The results indicate that students generally made a flexible use of both eliminative and confirmatory evaluation strategies when reading Internet search results, while some seemed to not pay attention to snippet and URL components of the search results.

Previous research indicates that web users rely to a great extent on the ranking provided by the search engine and predominantly access the first few web pages presented. In case that the information sources presented in the top of the search engine results page (SERP) are of rather low trustworthiness, this might lead to a biased or incomplete view of the topic—especially when dealing with controversial issues. Study 1, thus, systematically investigated whether participants who were asked to search for an unfamiliar and controversial medical issue accessed fewer trustworthy information sources and consequently included less information from trustworthy pages in their argumentation when the search results were ranked from least to most trustworthy on a Google-like SERP than when they were ranked from most to least trustworthy. Results from Study 1 confirmed these assumptions. Furthermore, Study 2 showed that when the same materials were presented in a grid interface, the impact of the position of the search results on their selection was substantially reduced. Irrespective of whether the most trustworthy search results were presented in the top or the bottom row of the grid interface, users predominantly selected the most trustworthy search results from the SERP and included the same amount of information from trustworthy pages in their argumentation.

In the above article, published online in Behaviour & Information Technology, http://dx.doi.org/10.1080/0144929X.2011.599040, Table 1 appeared incorrectly. The table should have appeared as follows...

In this paper, we present work in progress on how elementary school children use modern search engines to solve informational search tasks. Specifically, in a laboratory study with 21 children aged 8-10 we investigated whether the use of natural-language queries leads to more successful search outcomes than keyword queries when searching the Internet with Google. Both quantitative and qualitative data are reported that indicate the advantages of natural-language queries. Along, based on our observations we present a query-reformulation tool for a search engine interface for children that we are currently developing.

The present study examined how both the interface of search engines and Internet-specific epistemic beliefs influence novices' source evaluations during Web search on a medical topic. A standard Google-like list interface was compared to a tabular interface that presented search results grouped according to objective, subjective, or commercial information in order to provide users with affordances for source evaluations. Results revealed that university students using the tabular interface paid less visual attention to commercial search results and selected objective search results more often and commercial ones less often than students using the list interface. Furthermore, the epistemic belief that the Web contains (among other types of information) correct knowledge was related to an increased selection of objective search results and to longer fixations on non-selected search results. Moreover, epistemic beliefs moderated the effects of the search interface, such that students with strong beliefs that the Web contains correct knowledge showed a more focused information selection and better search outcomes in terms of their argumentative summaries when using the tabular interface than when using the list interface. In contrast, these effects were not found for students with doubts about the Web containing correct knowledge.

This paper presents a study which examined the selection of Web search results with a gaze-based input device. A standard list interface was compared to a grid and a tabular layout with regard to task performance and subjective ratings. Furthermore, the gaze-based input device was compared to conventional mouse interaction. Test persons had to accomplish a series of search tasks by selecting search results. The study revealed that mouse users accomplished more tasks correctly than users of the gaze-based input device. However, no differences were found between input devices regarding the number of search results taken into account to accomplish a task. Regarding task completion time and ease of search result selection only in the list interface gaze-based interaction was inferior to mouse interaction. Moreover, with a gaze-based input device search tasks were accomplished faster in tabular presentation than in a standard list interface, suggesting a tabular interface as best suited for gaze-based interaction.

This study examined to what extent users spontaneously evaluate the trustworthiness of Web search results presented by a search engine. For this purpose, a methodological paradigm was used in which the trustworthiness order of search results was experimentally manipulated by presenting search results on a search engine results page (SERP) either in a descending or ascending trustworthiness order. Moreover, a standard list format was compared to a grid format in order to examine the impact of the search results interface on Web users' evaluation processes. In an experiment addressing a controversial medical topic, 80 participants were assigned to one of four conditions with trustworthiness order (descending vs. ascending) and search results interface (list vs. grid) varied as between-subjects factors. In order to investigate participants' evaluation processes their eye movements and mouse clicks were captured during Web search. Results revealed that a list interface caused more homogenous and more linear viewing sequences on SERPs than a grid interface. Furthermore, when using a list interface most attention was given to the search results on top of the list. In contrast, with a grid interface nearly all search results on a SERP were attended to equivalently long. Consequently, in the ascending trustworthiness order participants using a list interface attended significantly longer to the least trustworthy search results and selected the most trustworthy search results significantly less often than participants using a grid interface. Thus, the presentation of Web search results by means of a grid interface seems to support users in their selection of trustworthy information sources.

Social tagging arose out of the need to organize found content that is worth revisiting. A significant side effect has been the use of social tagging sites as navigational signposts for interesting content. The collective behavior of users who tagged contents seems to offer a good basis for exploratory search interfaces, even for users who are not using social bookmarking sites. In this paper, we present the design of a tag-based exploratory system and detail an experiment in understanding its effectiveness. The tag-based search system allows users to utilize relevance feedback on tags to indicate their interest in various topics, enabling rapid exploration of the topic space. The experiment shows that the system seems to provide a kind of scaffold for users to learn new topics.

In this paper a study is reported, which investigates the effectiveness of two approaches to improving gaze-based interaction for realistic and complex menu selection tasks. The first approach focuses on identifying menu designs for hierarchical menus that are particularly suitable for gaze-based interaction, whereas the second approach is based on the idea of combining gaze-based interaction with speech as a second input modality. In an experiment with 40 participants the impact of menu design, input device, and navigation complexity on accuracy and completion time in a menu selection task as well as on user satisfaction were investigated. The results concerning both objective task performance and subjective ratings confirmed our expectations in that a semi-circle menu was better suited for gaze-based menu selection than either a linear or a full-circle menu. Contrary to our expectations, an input device solely based on eye gazes turned out to be superior to the combined gaze- and speech-based device. Moreover, the drawbacks of a less suitable menu design (i.e., of a linear menu or a full-circle menu) as well as of the multimodal input device particularly obstructed performance in the case of more complex navigational tasks.