Generative artificial intelligence (GenAI) tools, such as large language models (LLMs), generate natural language and other types of content to perform a wide range of tasks. This represents a significant technological advancement that poses opportunities and challenges to educational research and practice. This commentary brings together contributions from nine experts working in the intersection of learning and technology and presents critical reflections on the opportunities, challenges, and implications related to GenAI technologies in the context of education. In the commentary, it is acknowledged that GenAI’s capabilities can enhance some teaching and learning practices, such as learning design, regulation of learning, automated content, feedback, and assessment. Nevertheless, we also highlight its limitations, potential disruptions, ethical consequences, and potential misuses. The identified avenues for further research include the development of new insights into the roles human experts can play, strong and continuous evidence, human-centric design of technology, necessary policy, and support and competence mechanisms. Overall, we concur with the general skeptical optimism about the use of GenAI tools such as LLMs in education. Moreover, we highlight the danger of hastily adopting GenAI tools in education without deep consideration of the efficacy, ecosystem-level implications, ethics, and pedagogical soundness of such practices.

Augmented Reality (AR) tools have demonstrated considerable promise to enhance creative architectural design and support the retrofitting problem-solving processes through on-site daylighting visualization. AR’s capacity to integrate embodied motion enhances the non-expert’s understanding of the spatial characteristics and design ramifications within the built environment for complex facade design. Motion provides insights and increases the accessibility of retrofitting, encouraging more energy-efficient rework as opposed to complete building reconstruction. This study investigates the decision-making outcomes and cognitive-physical load implications of integrating a Building Information Modeling-driven AR system into the retrofitting design process and how movement is best leveraged to understand daylighting impacts. We conducted a study with 128 non-expert participants, who were asked to choose a window facade retrofit to improve an interior space. We analyze the effects of head movement, head rotations, and eye movements to understand how embodied motion improves overall objective performance across several daylighting and energy design metrics. We found no significant difference in the overall decision-making outcome between those who used an AR tool or a conventional desktop approach and that greater eye movement in AR was related to non-experts better balancing the complicated impacts facades have on daylight, aesthetics, and energy. This study indicates future expansion of AR retrofitting tools should encourage more eye movement.

Game-based learning environments have the distinctive capacity to promote learning experiences that are both engaging and effective. Recent advances in sensor-based technologies (e.g., facial expression analysis and eye gaze tracking) and natural language processing have introduced the opportunity to leverage multimodal data streams for learning analytics. Learning analytics and student modeling informed by multimodal data captured during students’ interactions with game-based learning environments hold significant promise for designing effective learning environments that detect unproductive student behaviors and provide adaptive support for students during learning. Learning analytics frameworks that can accurately predict student learning outcomes early in students’ interactions hold considerable promise for enabling environments to dynamically adapt to individual student needs. In this paper, we investigate a multimodal, multi-task predictive student modeling framework for game-based learning environments. The framework is evaluated on two datasets of game-based learning interactions from two student populations (n=61 and n=118) who interacted with two versions of a game-based learning environment for microbiology education. The framework leverages available multimodal data channels from the datasets to simultaneously predict student post-test performance and interest. In addition to inducing models for each dataset individually, this work investigates the ability to use information learned from one source dataset to improve models based on another target dataset (i.e., transfer learning using pre-trained models). Results from a series of ablation experiments indicate the differences in predictive capacity among a combination of modalities including gameplay, eye gaze, facial expressions, and reflection text for predicting the two target variables. In addition, multi-task models were able to improve predictive performance compared to single-task baselines for one target variable, but not both. Lastly, transfer learning showed promise in improving predictive capacity in both datasets.

Learning can be improved by providing personalized feedback adapting to the emotions that the learner may be experiencing. There is initial evidence that co-occurring emotions can be predicted during learning in Intelligent Tutoring Systems (ITS) through eye-tracking and interaction data. Predicting co-occurring emotions is a complex task and merging datasets has the potential to improve predictive performance. In this paper, we combine data from two user studies with an ITS, and analyze whether there is an improvement in predictive performance of co-occurring emotions, despite the user studies using different eye-trackers. In the pursuit towards developing real affect-aware ITS, we look at whether we can isolate classifiers that perform better than a baseline. In this regard we perform a series of statistical analyses and test out the predictive performance of standard machine learning models as well as an ensemble classifier for the task of predicting co-occurring emotions.

Augmented Reality (AR) tools have shown significant potential in providing on-site visualization of Building Information Modeling (BIM) data and models for supporting construction evaluation, inspection, and guidance. Retrofitting existing buildings, however, remains a challenging task requiring more innovative solutions to successfully integrate AR and BIM. This study aims to investigate the impact of AR+BIM technology on the retrofitting training process and assess the potential for future on-site usage. We conducted a study with 64 non-expert participants, who were asked to perform a common retrofitting procedure of an electrical outlet installation using either an AR+BIM system or a standard printed blueprint documentation set. Our findings indicate that AR+BIM reduced task time significantly and improved performance consistency across participants, while also decreasing the physical and cognitive demands of the training. This study provides a foundation for augmenting future retrofitting construction research that can extend the use of [Formula: see text] technology, thus facilitating more efficient retrofitting of existing buildings. A video presentation of this article and all supplemental materials are available at https://github.com/DesignLabUCF/SENSEable_RetrofittingTraining.

Modern text-to-speech engines can be an effective speech choice for embodied virtual pedagogical agents. However, it is not known how synthesized accents influence learning outcomes and perceptions of the agent. In this paper, we conducted a between-subjects experiment (n=60) to determine the effect of a pedagogical agent’s machine synthesized text-to-speech accent (United States English or Indian English) on learning outcomes and perceptions of the agent for students in the United States. Our results indicate that learner gender interacts with synthesized speech accent to significantly affect learning outcomes and perceptions of the agent. Our results reveal that a foreign synthetic speech accent may affect the learning outcomes of female university students (n=30), but not male university students (n=30). Finally, our results indicate that learner gender interacts with synthesized speech accent to affect perceptions of the pedagogical agent’s human-likeness. We provide novel insights on the differences between male and female learners for interactions with pedagogical agents with synthetic TTS accents.

Inability to regulate affective states can impact one's capacity to engage in higher-order thinking like scientific reasoning with game-based learning environments. Many efforts have been made to build affect-aware systems to mitigate the potentially detrimental effects of negative affect. Yet, gaps in research exist since accurately capturing and modeling affect as a state that changes dynamically over time is methodologically and analytically challenging. In this paper, we calculated multilevel mixed effects growth models to assess whether seventy-eight participants’ (n= 78) time engaging in scientific reasoning (via logfiles and eye gaze) were related to time facially expressing confused, frustrated, and neutral states (via facial recognition software) during game-based learning with Crystal Island. The fitted model estimated significant positive relations between the time learners facially expressed confusion, frustration, and neutral states and time engaging in scientific-reasoning actions. The time individual learners facially expressed frustrated, confused, and neutral states explained a significant amount of variation in time engaging in scientific reasoning. Our finding emphasize that individual differences and agency may play a important role on relations between affective states, their dynamics, and higher-order cognition during game-based learning. Designing affect-aware game-based learning environments that track the dynamics within individual learners’ affective states may best support cognition.

Significant efforts are currently being made to design affect-aware systems to classify, monitor, and scaffold emotional experiences across a range of settings. However, most investigations are limited in their view of emotions due to less sophisticated methodologies and analytical techniques. To address these issues, we captured 174 undergraduates’ emotions over time and defined them by multiple dimensions: (1) temporality, (2) valence, and (3) activation during learning with an intelligent tutoring system called MetaTutor. Latent growth models revealed the stability of negative activating emotions over time was related to performance, and changes in negative deactivating emotions were related to time engaging in cognitive strategies. Finally, a random forest classifier revealed high accuracy in predicting high (top 30%) and low performance groups (bottom 30%) using pre-test scores, changes in negative deactivating emotions, and time engaging in cognitive strategies. These findings have important implications for designing affect-aware systems that can potentially leverage emotion interventions based on if, when, and how an emotion changed (or remained stable) to optimize cognition and performance with emerging technologies.

Reflection plays a critical role in learning by encouraging students to contemplate their knowledge and previous learning experiences to inform their future actions and higher-order thinking, such as reasoning and problem solving. Reflection is particularly important in inquiry-driven learning scenarios where students have the freedom to set goals and regulate their own learning. However, despite the importance of reflection in learning, there are significant theoretical, methodological, and analytical challenges posed by measuring, modeling, and supporting reflection. This paper presents results from a classroom study to investigate middle-school students’ reflection during inquiry-driven learning with Crystal Island, a game-based learning environment for middle-school microbiology. To collect evidence of reflection during game-based learning, we used embedded reflection prompts to elicit written reflections during students’ interactions with Crystal Island. Results from analysis of data from 105 students highlight relationships between features of students’ reflections and learning outcomes related to both science content knowledge and problem solving. We consider implications for building adaptive support in game-based learning environments to foster deep reflection and enhance learning, and we identify key features in students’ problem-solving actions and reflections that are predictive of reflection depth. These findings present a foundation for providing adaptive support for reflection during game-based learning.

Modeling student knowledge is critical in adaptive learning environments. Predictive student modeling enables formative assessment of student knowledge and skills, and it drives personalized support to create learning experiences that are both effective and engaging. Traditional approaches to predictive student modeling utilize features extracted from students’ interaction trace data to predict student test performance, aggregating student test performance as a single output label. We reformulate predictive student modeling as a multi-task learning problem, modeling questions from student test data as distinct “tasks.” We demonstrate the effectiveness of this approach by utilizing student data from a series of laboratory-based and classroom-based studies conducted with a game-based learning environment for microbiology education, Crystal Island. Using sequential representations of student gameplay, results show that multi-task stacked LSTMs with residual connections significantly outperform baseline models that do not use the multi-task formulation. Additionally, the accuracy of predictive student models is improved as the number of tasks increases. These findings have significant implications for the design and development of predictive student models in adaptive learning environments.

Multimedia learning environments support learners in developing self-regulated learning (SRL) strategies. However, capturing these strategies and cognitive processes can be difficult for researchers because cognition is often inferred, not directly measured. This study sought to model self-reported metacognitive judgments using eye-tracking from 60 undergraduate students as they learned about biological systems with MetaTutorIVH, a multimedia learning environment. We found that participants’ gaze behaviors were different between the perceived relevance of the instructional content provided regardless of the actual content relevance. Additionally, we fit a cumulative link mixed effects ordinal regression model to explain reported metacognitive judgments based on content fixations, relevance, and presentation type. Main effects were found for all variables and several interactions between both fixations and content relevance as well as content fixations and presentation type. Surprisingly, accurate metacognitive judgments did not explain performance. Implication for multimedia learning environment design are discussed.

Recent advances in eye-tracking technologies have introduced the opportunity to incorporate gaze into student modeling. Creating student models that leverage gaze information holds significant promise for game-based learning environments. This paper introduces a gaze-enhanced student modeling framework that incorporates student eye tracking to dynamically predict students' performance in a game-based learning environment for microbiology education, CRYSTAL ISLAND. The gaze-enhanced student modeling framework was investigated in a study comparing a gaze-enhanced student model with a baseline student model that does not utilize student eye-tracking. Results of a study conducted with 65 college students interacting with the CRYSTAL ISLAND game-based learning environment indicate that the gaze-enhanced student model significantly outperforms the baseline model in dynamically predicting student problem-solving performance. The findings suggest that incorporating gaze into student modeling can contribute to a new generation of student models for game-based learning environments.

Affective support can play a central role in adaptive learning environments. Although virtual human tutors hold significant promise for providing affective support, a key open question is how a tutor's facial expressions can influence learners' performance. In this paper, we report on a study to examine the influence of a human tutor agent's facial expressions on learners' performance and emotions during learning. Results from the study suggest that learners' performance is significantly better when a human tutor agent facially expresses emotions that are congruent with the content relevancy. Results also suggest that learners facially express significantly more confusion when the human tutor agent provides incongruent facial expressions. These results can inform the design of virtual humans as pedagogical agents can inform the design of virtual humans as pedagogical agents and designing intelligent learner-agent interactions.

Emotions play a critical role during learning and problem solving with advanced learning technologies (ALTs). Despite their importance, relatively few attempts have been made to understand learners' emotional monitoring and regulation by using data visualizations of their own (and others') cognitive, affective, metacognitive, and motivational (CAMM) self-regulated learning (SRL) processes to potentially foster their emotion regulation (ER). We present a theoretically based and empirically driven conceptual framework that addresses ER by proposing the use of visualizations of one's own and others' CAMM SRL multichannel data to facilitate learners' monitoring and regulation of emotions during learning with ALTs. We use an example with eye-tracking data to illustrate the mapping between theoretical assumptions, ER strategies, and the types of data visualizations that can enhance learners' ER, including key processes such as emotion flexibility, emotion adaptivity, and emotion efficacy. We conclude with future directions leading to a systematic interdisciplinary research agenda that addresses outstanding ER-related issues by integrating models, theories, methods, and analytical techniques for the cognitive, learning, and affective sciences; human- computer interaction (HCI); data visualization; big data; data mining; and SRL.

In this poster, we describe the aim and current activities of the EARLI-Centre for Innovative Research (E-CIR) "Measuring and Supporting Student's Self-Regulated Learning in Adaptive Educational Technologies" which is funded by the European Association for Research on Learning and Instruction (EARLI) from 2015 to 2019. The aim is to develop our understanding of multimodal data that unobtrusively capture cognitive, meta-cognitive, affective and motivational states of learners over time. This demands for a concerted interdisciplinary dialogue combining findings from psychology and educational sciences with advances in computer sciences and artificial intelligence. The participants in this E-CIR are leading international researchers who have articulated different emerging perspectives and methodologies to measure cognition, metacognition, motivation, and emotions during learning. The participants recognize the need for intensive collaboration to accelerate progress with new interdisciplinary methods including learning analytics to develop more powerful adaptive educational technologies.

Self-regulated learning (SRL) is a process that highly fluctuates as students actively deploy their metacognitive and cognitive processes during learning. In this paper, we apply an extension of latent profiling, latent transition analysis (LTA), which investigates the longitudinal development of students' SRL latent class memberships over time. We will briefly review the theoretical foundations of SRL and discuss the value of using LTA to investigate this multidimensional concept. This study is based on college students (n = 75) learning about the human circulatory system while using MetaTutor, an intelligent tutoring system that adaptively supports SRL and targets specific metacognitive SRL processes including judgment of learning (JOL) and content evaluation (CE). Preliminary results identify transitional probabilities of SRL profiles from four distinct events associated with the use of SRL.

Recent years have seen a growing recognition of the role that affect plays in learning. Because game-based learning environments elicit a wide range of student affective states, affect-enhanced student modeling for game-based learning holds considerable promise. This paper introduces an affect-enhanced student modeling framework that leverages facial expression tracking for game-based learning. The affect-enhanced student modeling framework was used to generate predictive models of student learning and student engagement for students who interacted with CRYSTAL ISLAND, a game-based learning environment for microbiology education. Findings from the study reveal that the affect-enhanced student models significantly outperform baseline predictive student models that utilize the same gameplay traces but do not use facial expression tracking. The study also found that models based on individual facial action coding units are more effective than composite emotion models. The findings suggest that introducing facial expression tracking can improve the accuracy of student models, both for predicting student learning gains and also for predicting student engagement.

In self-regulated learning concept, Intelligent Tutoring Systems (ITS) can be designed to foster learning behaviors through pedagogical agents (PAs) that are used for interactions and exchange information with the human learner. These agents are intelligent and follow rational behaviors, but in the case of multi-agent environments they need to be systematically and specifically designed, however in order to follow a common goal, different self-regulatory systems have been designed that use pedagogical agents, but they fail to constrain the decision making of the agents and maintain a sequential decision making process during learning interactions with human learners. In this paper, we provide a new theoretical model for agent-learner interactions in MetaTutor, a multi-agent hypermedia learning environment, using Markovdecision processes. We theoretically define the agents' Markovdecisions and their influence on MetaTutor's performance as a whole. First, we formally define the Markov architecture and its parameters. We then link these characteristics to the pedagogical agents we use in MetaTutor and define different versions of MetaTutor agents equipped with Markov decision mechanism. Furthermore, we explore additional details about agents' sequential decision making and how reward functions influence their acting strategies with learners in the learning environment. We introduce the optimization problem in which we aim to maximize the expected return of the overall agents' acts in a self-regulatory system. What specifically distinguishes this work from the previous proposals in the same domain is its novelty in continuous decision making mechanism investigation and performance analysis that improve the applicability of the proposed adaptive model in a multi-agent ITS like MetaTutor.

We report preliminary data of an initial laboratory study examining the effectiveness of self-regulated learning (SRL) training versus no training on learners' ability to deploy SRL processes and learn about the circulatory system with MetaTutor. MetaTutor is an intelligent tutoring system (ITS) designed to train and foster learners' SRL processes while learning about several complex human body systems. We used a mixed methodology approach and include the results of a subset of the participants (N=30) whose product and process data we have analyzed. Overall, the results indicate that the SRL training group significantly outperformed the control group.

This study investigated the navigation patterns of 56 learners within a hypermedia learning environment, MetaTutor. Using K-Means cluster analysis, four types of navigational profiles were created. One cluster including participants who navigated primarily linearly through the learning environment, one showed high levels of non-linear progression, one included participants who opened the images very frequently, and the last cluster had a balance of all the navigational variables included in the analysis. Data from the learning measures used indicated that the balanced cluster and the cluster which opened the image frequently had the highest learning outcomes. Implications for the design of adaptive hypermedia learning systems are discussed.