Maintenance of medium voltage switchgears is a complex task that requires extensive expertise. However, a substantial portion of this knowledge is often difficult for the next generation of field technicians to access and apply effectively. For instance, maintenance reports – which document system issues and corrective actions – represent a valuable knowledge source yet frequently underutilized due to their unstructured and sparse nature. In this context, recent advancements in Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) techniques have been explored to analyze historical maintenance reports and provide real-time guidance to field technicians. However, chatbot-based interactions remain constrained by the stochastic nature of responses, challenges related to domain and industry specificity, and risks of hallucination. This study introduces a novel approach that leverages LLMs to systematically extract structured, machine-readable rules – i.e., sequences of actions used to resolve specific issues – directly from maintenance reports. A human-in-the-loop approach is proposed, where a human expert validates each extracted rule, mitigating inaccuracies and enhancing quality control. These validated rules are eventually stored in a dedicated rule corpus and later deployed in a deterministic rule engine, ensuring reliability and consistency. An industrial case study evaluates the effectiveness of this approach by comparing LLM-generated rules with those manually curated by domain experts. By shifting from conventional chatbot-like interaction to a more human-centric structured rule extraction, our methodology enhances reproducibility, reliability, and accessibility in industrial maintenance knowledge management. This approach addresses hallucinations and increases robustness of LLM applications in industrial AI settings.

Web-based learning is evolving rapidly as traditional search engines are complemented by Large Language Models (LLMs) and other AI technologies. This evolution offers new opportunities, such as automated information synthesis and personalized learning experiences. However, this also presents new challenges, including the need for learners to be aware of potential biases and misinformation in AI-generated content, and to maintain focus and depth in their learning journeys.

Misinformation is an important topic in the Information Retrieval (IR) context and has implications for both system-centered and user-centered IR. While it has been established that the performance in discerning misinformation is affected by a person’s cognitive load, the variation in cognitive load in judging the veracity of news is less understood. To understand the variation in cognitive load imposed by reading news headlines related to COVID-19 claims, within the context of a fact-checking system, we conducted a within-subject, lab-based, quasi-experiment (N=40) with eye-tracking. Our results suggest that examining true claims imposed a higher cognitive load on participants when news headlines provided incorrect evidence for a claim and were inconsistent with the person’s prior beliefs. In contrast, checking false claims imposed a higher cognitive load when the news headlines provided correct evidence for a claim and were consistent with the participants’ prior beliefs. However, changing beliefs after examining a claim did not have a significant relationship with cognitive load while reading the news headlines. The results illustrate that reading news headlines related to true and false claims in the fact-checking context impose different levels of cognitive load. Our findings suggest that user engagement with tools for discerning misinformation needs to account for the possible variation in the mental effort involved in different information contexts.

We conducted a lab-based eye-tracking study to investigate how interactivity of an AI-powered fact-checking system affects user interactions, such as dwell time, attention, and mental resources involved in using the system. A within-subject experiment was conducted, where participants used an interactive and a non-interactive version of a mock AI fact-checking system, and rated their perceived correctness of COVID-19 related claims. We collected web-page interactions, eye-tracking data, and mental workload using NASA-TLX. We found that the presence of the affordance of interactively manipulating the AI system's prediction parameters affected users’ dwell times, and eye-fixations on AOIs, but not mental workload. In the interactive system, participants spent the most time evaluating claims’ correctness, followed by reading news. This promising result shows a positive role of interactivity in a mixed-initiative AI-powered system.

An important step to foster learning during search is to identify behavioural patterns that distinguish searchers gaining more vs. less knowledge during search. Previous efforts have studied searchers in the short term, typically during a single lab session. We propose a longitudinal study to analyse searching behaviour of students enrolled in a university course, over the span of a few months. Our research aims are to (i) identify how searching behaviour changes over time, as students gain new knowledge on a subject, (ii) whether the perceived-relevance of specific information objects change due to knowledge acquisition, and, (iii) identify if differences exist in search processes for previously seen vs. unseen search tasks. We posit that findings from this research study will be informative for building improved information retrieval systems which are supportive of search as learning.

Collecting participant search logs is an integral part of interactive IR research. Today's existing approaches are either piecemeal solutions, and/or require cumbersome setups. We present YASBIL, a two-component logging solution comprising a browser extension and a WordPress plugin. The browser extension logs the browsing activity in the participants' machines. The WordPress plugin collects the logged data into the researcher's data server. The logging works on any webpage, without the need to own or have knowledge about the HTML structure of the webpage. YASBIL also offers ethical data transparency and security towards participants, by enabling them to view and obtain copies of the logged data, as well as securely upload the data to the researcher's server over an HTTPS connection. We posit that ease of installation and use will make YASBIL especially suitable for remote user-studies, and longitudinal studies in IR.

We propose an image-classification method to predict the perceived-relevance of text documents from eye-movements. An eye-tracking study was conducted where participants read short news articles, and rated them as relevant or irrelevant for answering a trigger question. We encode participants' eye-movement scanpaths as images, and then train a convolutional neural network classifier using these scanpath images. The trained classifier is used to predict participants' perceived-relevance of news articles from the corresponding scanpath images. This method is content-independent, as the classifier does not require knowledge of the screen-content, or the user's information-task. Even with little data, the image classifier can predict perceived-relevance with up to 80% accuracy. When compared to similar eye-tracking studies from the literature, this scanpath image classification method outperforms previously reported metrics by appreciable margins. We also attempt to interpret how the image classifier differentiates between scanpaths on relevant and irrelevant documents.

The task of answering questions about images has garnered attention as a practical service for assisting populations with visual impairments as well as a visual Turing test for the artificial intelligence community. Our first aim is to identify the common vision skills needed for both scenarios. To do so, we analyze the need for four vision skills--object recognition, text recognition, color recognition, and counting--on over 27,000 visual questions from two datasets representing both scenarios. We next quantify the difficulty of these skills for both humans and computers on both datasets. Finally, we propose a novel task of predicting what vision skills are needed to answer a question about an image. Our results reveal (mis)matches between aims of real users of such services and the focus of the AI community. We conclude with a discussion about future directions for addressing the visual question answering task.

We investigate the relationship between search behavior, eye -tracking measures, and learning. We conducted a user study where 30 participants performed searches on the web. We measured their verbal knowledge before and after each task in a content-independent manner, by assessing the semantic similarity of their entries to expert vocabulary. We hypothesize that differences in verbal knowledge-change of participants are reflected in their search behaviors and eye-gaze measures related to acquiring information and reading. Our results show that participants with higher change in verbal knowledge differ by reading significantly less, and entering more sophisticated queries, compared to those with lower change in knowledge. However, we do not find significant differences in other search interactions like page visits, and number of queries.

The complex stochastic nature of eye tracking data calls for exploring sophisticated statistical models to ensure reliable inference in multi-trial eye-tracking experiments. We employ a Bayesian semi-parametric mixed-effects Markov model to compare gaze transition matrices between different experimental factors accommodating individual random effects. The model not only allows us to assess global influences of the external factors on the gaze transition dynamics but also provides comprehension of these effects at a deeper local level. We experimented to explore the impact of recognizing distorted images of artwork and landmarks on the gaze transition patterns. Our dataset comprises sequences representing areas of interest visited when applying a content independent grid to the resulting scan paths in a multi-trial setting. Results suggest that image recognition to some extent affects the dynamics of the transitions while image type played an essential role in the viewing behavior.

Visual question answering is the task of returning the answer to a question about an image. A challenge is that different people often provide different answers to the same visual question. To our knowledge, this is the first work that aims to understand why. We propose a taxonomy of nine plausible reasons, and create two labelled datasets consisting of ~45,000 visual questions indicating which reasons led to answer differences. We then propose a novel problem of predicting directly from a visual question which reasons will cause answer differences as well as a novel algorithm for this purpose. Experiments demonstrate the advantage of our approach over several related baselines on two diverse datasets. We publicly share the datasets and code at https://vizwiz.org.

We conducted an eye-tracking study where 30 participants performed searches on the web. We measured their topical knowledge before and after each task. Their eye-fixations were labelled as "reading" or "scanning". The series of reading fixations in a line, called "reading-sequences" were characterized by their length in pixels, fixation duration, and the number of fixations making up the sequence. We hypothesize that differences in knowledge-change of participants are reflected in their eye-tracking measures related to reading. Our results show that the participants with higher change in knowledge differ significantly in terms of their total reading-sequence-length, reading-sequence-duration, and number of reading fixations, when compared to participants with lower knowledge-change.