This article presents a methodology for advancing Learning Analytics by automatically reconstructing the organisational structure of Moodle courses leveraging platform logs and backup data. Using an extensive, pseudonymised dataset gathered from twelve undergraduate courses, we illustrate how student activity records can be enriched with structural information. This methodology integrates xAPI statements with parsed XML data from Moodle course backups to create directed graph representations that capture both hierarchical and sequential relationships among sections and activities. By introducing graph-based structural modelling into Learning Analytics practice, we provide new tools to facilitate replicable and context-aware analytics, enabling deeper insights into the relationship between course structure and learning behaviours, as well as actionable enhancement of digital course design.

International audience

International audience

International audience

Adaptive support can help learners in Open-Ended Learning Environments (OELEs), where the free-form nature of the interaction can be confusing to students. In this paper, we design and evaluate an Intelligent Pedagogical Agent (IPA) for an OELE designed to foster Computational Thinking (CT). Specifically, we design help interventions for an in-the-wild scenario where students interact with the OELE in an unmonitored, self-directed manner. We build a student model by extracting meaningful student behaviors on real-world interaction data obtained during interaction in online classrooms and including expert insights. We show that these student models perform better than a baseline and have the potential for adaptive support in self-directed interaction with the OELE. We design an IPA with the help of teachers, leveraging the student behaviors extracted from data. Lastly, we get insights into the value of these help interventions by empirically evaluating the IPA in a formal user study.

Research has shown that free-form Game-Design (GD) environments can be very effective in fostering Computational Thinking (CT) skills at a young age. However, some students can still need some guidance during the learning process due to the highly open-ended nature of these environments. Intelligent Pedagogical Agents (IPAs) can be used to provide personalized assistance in real-time to alleviate this challenge. This paper presents our results in evaluating such an agent deployed in a real-word free-form GD learning environment to foster CT in the early K-12 education, Unity-CT. We focus on the effect of repetition by comparing student behaviors between no intervention, 1-shot, and repeated intervention groups for two different errors that are known to be challenging in the online lessons of Unity-CT. Our findings showed that the agent was perceived very positively by the students and the repeated intervention showed promising results in terms of helping students make fewer errors and more correct behaviors, albeit only for one of the two target errors. Building from these results, we provide insights on how to provide IPA interventions in free-form GD environments.

Free-form Game-Design (GD) environments show promise in fostering Computational Thinking (CT) skills at a young age. However, such environments can be challenging to some students due to their highly open-ended nature. Our long-term goal is to alleviate this difficulty via pedagogical agents that can monitor the student interaction with the environment, detect when the student needs help and provide personalized support accordingly. In this paper, we present a preliminary evaluation of one such agent deployed in a real-word free-form GD learning environment to foster CT in the early K-12 education, Unity-CT. We focus on the effect of repetition by comparing student behaviors between no intervention, 1-shot, and repeated intervention groups for two different errors that are known to be challenging in the online lessons of Unity-CT environment. Our findings showed that the agent was perceived very positively by the students and the repeated intervention showed promising results in terms of helping students make less errors and more correct behaviors, albeit only for one of the two target errors. Based on these results, we provide insights on how to improve the delivery of the agent’s interventions in free-form GD environments.

Game-Design (GD) environments show promise in fostering Computational Thinking (CT) skills at a young age. However, such environments can be challenging to some students due to their highly open-ended nature. We propose to alleviate this difficulty by learning interpretable student models from data that can drive personalization of a real-world GD learning environment to the student’s needs. We apply our approach on a dataset collected in ecological settings and evaluate the ability of the generated student models at predicting ineffective learning behaviors over the course of the interaction. We then discuss how these behaviors can be used to define personalized support in GD learning activities, by conducting extensive interviews with experienced instructors.

We study the effectiveness of adaptive interventions at helping users process textual documents with embedded visualizations, a form of multimodal documents known as Magazine-Style Narrative Visualizations (MSNVs). The interventions are meant to dynamically highlight in the visualization the datapoints that are described in the textual sentence currently being read by the user, as captured by eye-tracking. These interventions were previously evaluated in two user studies that involved 98 participants reading excerpts of real-world MSNVs during a 1-hour session. Participants’ outcomes included their subjective feedback about the guidance, and well as their reading time and score on a set of comprehension questions. Results showed that the interventions can increase comprehension of the MSNV excerpts for users with lower levels of a cognitive skill known as visualization literacy. In this article, we aim to further investigate this result by leveraging eye-tracking to analyze in depth how the participants processed the interventions depending on their levels of visualization literacy. We first analyzed summative gaze metrics that capture how users process and integrate the key components of the narrative visualizations. Second, we mined the salient patterns in the users’ scanpaths to contextualize how users sequentially process these components. Results indicate that the interventions succeed in guiding attention to salient components of the narrative visualizations, especially by generating more transitions between key components of the visualization (i.e., datapoints, labels, and legend), as well as between the two modalities (text and visualization). We also show that the interventions help users with lower levels of visualization literacy to better map datapoints to the legend, which likely contributed to their improved comprehension of the documents. These findings shed light on how adaptive interventions help users with different levels of visualization literacy, informing the design of personalized narrative visualizations.

In this article, we investigate the value of gaze-driven adaptive interventions to support the processing of textual documents with embedded visualizations, i.e., Magazine Style Narrative Visualizations (MSNVs). These interventions are provided dynamically by highlighting relevant data points in the visualization when the user reads related sentences in the MSNV text, as detected by an eye-tracker. We conducted a user study during which participants read a set of MSNVs with our interventions, and compared their performance and experience with participants who received no interventions. Our work extends previous findings by showing that dynamic, gaze-driven interventions can be delivered based on reading behaviors in MSNVs, a widespread form of documents that have never been considered for gaze-driven adaptation so far. Next, we found that the interventions significantly improved the performance of users with low levels of visualization literacy, i.e., those users who need help the most due to their lower ability to process and understand data visualizations. However, high literacy users were not impacted by the interventions, providing initial evidence that gaze-driven interventions can be further improved by personalizing them to the levels of visualization literacy of their users.

We leverage eye-tracking data to predict user performance and levels of cognitive abilities while reading magazine-style narrative visualizations (MSNV), a widespread form of multimodal documents that combine text and visualizations. Such predictions are motivated by recent interest in devising user-adaptive MSNVs that can dynamically adapt to a user's needs. Our results provide evidence for the feasibility of real-time user modeling in MSNV, as we are the first to consider eye tracking data for predicting task comprehension and cognitive abilities while processing multimodal documents. We follow with a discussion on the implications to the design of personalized MSNVs.

We study the effectiveness of adaptive guidance at helping users process textual documents with embedded visualizations, known as narrative visualizations. We do so by leveraging eye tracking to analyze in depth the effect that adaptations meant to guide the user's gaze to relevant parts of the visualizations has on users with different levels of visualization literacy. Results indicate that the adaptations succeed in guiding attention to salient components of the narrative visualizations, especially by generating more transitions between key components of the visualization (i.e., datapoints, labels and legend). We also show that the adaptation helps users with lower levels of visualization literacy to better map datapoints to the legend, which leads in part to improved comprehension of the visualization. These findings shed light on how adaptive guidance helps users with different levels of visualization literacy, informing the design of personalized narrative visualizations.

Previous work has shown that some user cognitive abilities relevant for processing information visualizations can be predicted from eye-tracking data. Performing this type of user modeling is important for devising visualizations that can detect a user's abilities and adapt accordingly during the interaction. In this article, we extend previous user modeling work by investigating for the first time interaction data as an alternative source to predict cognitive abilities during visualization processing when it is not feasible to collect eye-tracking data. We present an extensive comparison of user models based solely on eye-tracking data, on interaction data, as well as on a combination of the two. Although we found that eye-tracking data generate the most accurate predictions, results show that interaction data can still outperform a majority-class baseline, meaning that adaptation for interactive visualizations could be enabled even when it is not feasible to perform eye tracking, using solely interaction data. Furthermore, we found that interaction data can predict several cognitive abilities with better accuracy at the very beginning of the task than eye-tracking data, which are valuable for delivering adaptation early in the task. We also extend previous work by examining the value of multimodal classifiers combining interaction data and eye-tracking data, with promising results for some of our target user cognitive abilities. Next, we contribute to previous work by extending the type of visualizations considered and the set of cognitive abilities that can be predicted from either eye-tracking data and interaction data. Finally, we evaluate how noise in gaze data impacts prediction accuracy and find that retaining rather noisy gaze datapoints can yield equal or even better predictions than discarding them, a novel and important contribution for devising adaptive visualizations in real settings where eye-tracking data are typically noisier than in laboratory settings.

We present an experimenter platform for designing and evaluating user-adaptive support in information visualizations. Specifically, this platform leverages eye-tracking data in real time to deliver adaptive support in visualizations based on the user's attentional patterns and individual needs. We describe the main functionalities of this platform, and show an application to support processing of textual documents with embedded bar charts, by dynamically providing highlighting in the charts to guide a user's attention to the relevant information.

Although there is extensive evidence that personalization of interactive systems can improve the user's experience and satisfaction, it is also known that the two main approaches to deliver personalization, namely via customization or system-driven adaptation, have limitations. In particular, many users do not use customize mechanisms, while adaptation can be perceived as intrusive and opaque. In this paper, we explore an intermediary approach to personalization, namely delivering system-driven support to customization. To this end, we study a customization mechanism allowing to choose the type and amount of information displayed by means of information visualizations in a system for decision making, and examine the impact of user differences on the effectiveness of this mechanism. Our results show that, for the users who did use the customization mechanism, customization effectiveness was impacted by their levels of visualization literacy and locus of control. These results suggest that the customization mechanism could be improved by system-driven assistance to customize depending on the user's level of visualization literacy and locus of control.

Previous work has shown that some user cognitive abilities relevant for processing information visualizations can be predicted from eye tracking data. Performing this type of user modeling is important for devising user-adaptive visualizations that can adapt to a user’s abilities as needed during the interaction. In this paper, we contribute to previous work by extending the type of visualizations considered and the set of cognitive abilities that can be predicted from gaze data, thus providing evidence on the generality of these findings. We also evaluate how quality of gaze data impacts prediction.

In this paper we investigate using a variety of behavioral measures collectible with an eye tracker to predict a user's skill acquisition phase while performing various information visualization tasks with bar graphs. Our long term goal is to use this information in real-time to create user-adaptive visualizations that can provide personalized support to facilitate visualization processing based on the user's predicted skill level. We show that leveraging two additional content-independent data sources, namely information on a user's pupil dilation and head distance to the screen, yields a significant improvement for predictive accuracies of skill acquisition compared to predictions made using content-dependent information related to user eye gaze attention patterns, as was done in previous work. We show that including features from both pupil dilation and head distance to the screen improve the ability to predict users' skill acquisition state, beating both the baseline and a model using only content-dependent gaze information.

There is increasing evidence that the effectiveness of Information Visualization (Infovis) is affected by the user needs and abilities. For instance, cognitive abilities (e.g., perceptual speed, working memory) [e.g., 1-4] have been shown to impact users' performance and satisfaction with a given visualization. These findings suggest that it can be valuable to develop visualization systems that can provide personalized support targeting specific user characteristics. Furthermore, recent research [e.g., 3,5] has shown that eye tracking data can be leveraged to identify the elements of a visualization for which specific user differences hinder user experience or performance, thus providing concrete information on which specific personalized support could be helpful for different users (e.g., users with low perceptual speed may benefit from help in processing legends [1]). Though these findings are encouraging toward the design of user-adaptive or customized visualizations, they are generally related to either fictional tasks or research prototypes. So, it is unclear if existing results on the value of user-adaptive visualizations can transfer to real-world settings.

This paper summarizes an ongoing multi-year project aiming to uncover knowledge and techniques for devising intelligent environments for user-adaptive visualizations. We ran three studies designed to investigate the impact of user and task characteristics on user performance and satisfaction in different visualization contexts. Eye-tracking data collected in each study was analyzed to uncover possible interactions between user/task characteristics and gaze behavior during visualization processing. Finally, we investigated user models that can assess user characteristics relevant for adaptation from eye tracking data.

User performance and satisfaction when working with an interface is influenced by how quickly the user can acquire the skills necessary to work with the interface through practice. Learning curves are mathematical models that can represent a user's skill acquisition ability through parameters that describe the user's initial expertise as well as her learning rate. This information could be used by an interface to provide adaptive support to users who may otherwise be slow in learning the necessary skills. In this paper, we investigate the feasibility of predicting in real time a user's learning curve when working with ValueChart, an interactive visualization for decision making. Our models leverage various data sources (a user's gaze behavior, pupil dilation, cognitive abilities), and we show that they outperform a baseline that leverages only knowledge on user task performance so far. We also show that the best performing model achieves good accuracies in predicting users' learning curves even after observing users' performance only on a few tasks. These results are promising toward the design of user-adaptive visualizations that can dynamically support a user in acquiring the necessary skills to complete visual tasks.