We present an empirical study examining how experienced tutors (experts) and non-tutors (novices) evaluate the correctness of tutor praise responses under different AI-assisted decision-support interfaces and explanation styles. We examine human-AI reliance patterns by decomposing interaction errors into over-reliance (accepting incorrect AI suggestions) and under-reliance (rejecting correct AI suggestions), together with time cost as a process-level indicator. Across conditions, human-AI collaboration improved accuracy compared to humans working alone, but consistently underperformed an AI-only baseline, indicating that human judgment introduced additional errors even when assisted by a highly accurate model. Novices benefited more from AI support since they tend to follow AI suggestions, whereas experts frequently overrode correct AI advice, resulting in lower overall performance, revealing a paradox of expertise in educational decision-making. We further compare two explanation modalities: textual reasoning and inline highlighting. Textual reasoning reduced under-reliance when the AI was correct but increased over-reliance when the AI was wrong, while inline highlighting exerted minimal influence on either behavior. Notably, neither explanation modality improved accuracy, and both increased time costs. As a contribution to learning analytics, we demonstrate how reliance patterns (over-reliance and under-reliance) and time cost function as process-level indicators that reveal how users integrate, or fail to integrate, AI recommendations. Our findings underscore the need for adaptive, trust-calibrated explanation strategies in tutor-facing decision support systems that balance accuracy, efficiency, and accountability in human-AI collaboration.

Providing timely, targeted, and multimodal feedback helps students quickly correct errors, build deep understanding and stay motivated, yet making it at scale remains a challenge. This study introduces a real-time AI-facilitated multimodal feedback system that integrates structured textual explanations with dynamic multimedia resources, including the retrieved most relevant slide page references and streaming AI audio narration. In an online crowdsourcing experiment, we compared this system against fixed business-as-usual feedback by educators across three dimensions: (1) learning effectiveness, (2) learner engagement, (3) perceived feedback quality and value. Results showed that AI multimodal feedback achieved learning gains equivalent to original educator feedback while significantly outperforming it on perceived clarity, specificity, conciseness, motivation, satisfaction, and reducing cognitive load, with comparable correctness, trust, and acceptance. Process logs revealed distinct engagement patterns: for multiple-choice questions, educator feedback encouraged more submissions; for open-ended questions, AI-facilitated targeted suggestions lowered revision barriers and promoted iterative improvement. These findings highlight the potential of AI multimodal feedback to provide scalable, real-time, and context-aware support that both reduces instructor workload and enhances student experience.

Student-risk prediction models are often proposed as tools for helping higher education institutions identify students who may need additional advising or teaching support. In resource-constrained settings, these models may be used to rank students by predicted risk so that limited support can be prioritized for those at the top of the list. However, fairness is commonly evaluated by applying a single decision threshold to risk scores and comparing classification errors across demographic groups. This creates a mismatch: a model may appear fairer under threshold-based evaluation while producing the same ranked support list used for intervention. We examine this mismatch by evaluating the same risk scores under two decision settings: threshold-based fairness, which measures group disparities in classification outcomes at a decision threshold, and allocation fairness, which assesses whether students selected in a capacity-limited top-\(K\) support list reflect observed need across groups. Using two public higher-education datasets, we compare threshold-based fairness reports with top-\(K\) selection under fixed support capacity. Results show that improvements in threshold-based fairness metrics do not necessarily change who is selected for support. In particular, threshold adjustment can reduce false-negative-rate gaps without changing the top-\(K\) list, while disability-related under-selection relative to observed need remains visible in OULAD. These findings suggest that fairness evaluation for student-risk prediction should report both threshold-based disparities and whether ranked support lists align with observed need across groups.

Many edtech systems use a knowledge-mastery threshold to decide when learners can stop practising a skill and move on to new content. However, learners can cross this threshold even when their recent traces show weak self-regulatory behaviours, such as rapid guessing or getting answers right only after repeated attempts. This raises concern that these learners are likely to struggle on subsequent work once practice is stopped, even though the system has already advanced them. To address this decision risk, our study proposes an approach that augments standard knowledge tracing (KT) mastery predictions with trace-based self-regulated learning (SRL) indicators, using ASSISTments Skill Builder 2009--2010 data logs. Specifically, we adopt a trace-based SRL framework and define weak regulation at the mastery boundary as low-quality engagement around the moment when the system first declares a skill ''mastered'', operationalised via behavioural indicators derived from learners' recent practice traces. We then test whether KT mastery crossings that co-occur with weak-regulation indicators predict less stable performance after the system ends practice on the skill, including higher rates of later performance drops and slower recovery after errors. Our results show that these ''KT-mastered but weakly regulated'' cases are common across skills and consistently associated with poorer subsequent outcomes than other KT-mastered cases. Together, the findings motivate adding lightweight SRL checks at the mastery boundary to improve the reliability of mastery-based progression decisions, especially in learning-at-scale settings where support resources are limited.

The role of multiple-choice questions (MCQs) as effective learning tools has been debated in past research. While MCQs are widely used due to their ease in grading, open response questions are increasingly used for instruction, given advances in large language models (LLMs) for automated grading. This study evaluates MCQs effectiveness relative to open-response questions, both individually and in combination, on learning. These activities are embedded within six tutor lessons on advocacy. Using a posttest-only randomized control design, we compare the performance of 234 tutors (790 lesson completions) across three conditions: MCQ only, open response only, and a combination of both. We find no significant learning differences across conditions at posttest, but tutors in the MCQ condition took significantly less time to complete instruction. These findings suggest that MCQs are as effective, and more efficient, than open response tasks for learning when practice time is limited. To further enhance efficiency, we autograded open responses using GPT-4o and GPT-4-turbo. GPT models demonstrate proficiency for purposes of low-stakes assessment, though further research is needed for broader use. This study contributes a dataset of lesson log data, human annotation rubrics, and LLM prompts to promote transparency and reproducibility.

Equity is a core concern of learning analytics. However, applications that teach and assess equity skills, particularly at scale are lacking, often due to barriers in evaluating language. Advances in generative AI via large language models (LLMs) are being used in a wide range of applications, with this present work assessing its use in the equity domain. We evaluate tutor performance within an online lesson on enhancing tutors' skills when responding to students in potentially inequitable situations. We apply a mixed-method approach to analyze the performance of 81 undergraduate remote tutors. We find marginally significant learning gains with increases in tutors' self-reported confidence in their knowledge in responding to middle school students experiencing possible inequities from pretest to posttest. Both GPT-4o and GPT-4-turbo demonstrate proficiency in assessing tutors ability to predict and explain the best approach. Balancing performance, efficiency, and cost, we determine that few-shot learning using GPT-4o is the preferred model. This work makes available a dataset of lesson log data, tutor responses, rubrics for human annotation, and generative AI prompts. Future work involves leveling the difficulty among scenarios and enhancing LLM prompts for large-scale grading and assessment.

Caregivers (i.e., parents and members of a child's caring community) are underappreciated stakeholders in learning analytics. Although caregiver involvement can enhance student academic outcomes, many obstacles hinder involvement, most notably knowledge gaps with respect to modern school curricula. An emerging topic of interest in learning analytics is hybrid tutoring, which includes instructional and motivational support. Caregivers assert similar roles in homework, yet it is unknown how learning analytics can support them. Our past work with caregivers suggested that conversational support is a promising method of providing caregivers with the guidance needed to effectively support student learning. We developed a system that provides instructional support to caregivers through conversational recommendations generated by a Large Language Model (LLM). Addressing known instructional limitations of LLMs, we use instructional intelligence from tutoring systems while conducting prompt engineering experiments with the open-source Llama 3 LLM. This LLM generated message recommendations for caregivers supporting their child's math practice via chat. Few-shot prompting and combining real-time problem-solving context from tutoring systems with examples of tutoring practices yielded desirable message recommendations. These recommendations were evaluated with ten middle school caregivers, who valued recommendations facilitating content-level support and student metacognition through self-explanation. We contribute insights into how tutoring systems can best be merged with LLMs to support hybrid tutoring settings through conversational assistance, facilitating effective caregiver involvement in tutoring systems.

published_or_final_version

Broad Al literacy is essential in today's rapidly advancing technological landscape, extending beyond Al specialists to encompass the general public. However, the complexity of Al concepts poses significant barriers to learning for individuals without prior Al knowledge. While teaching through analogies is a well-recognized method to simplify complex information by connecting it to familiar concepts, adapting these analogies to match individual learner profiles remains a substantial challenge. This paper addresses this gap by proposing a novel method for personalizing educational analogies, enhancing the accessibility and engagement of AI concepts for a diverse audience. Our approach uses Large language models (LLMs) to dynamically tailor content to each learner's cognitive and cultural contexts, grounded in educational theories and practices. Utilizing a crowdsourced AIB testing framework through Prolific (N-60), this research contrasts conventional instructional methods with content incorporating LLM-enhanced personalized analogies. Data collection comprised pre- and post-tests, activity logs, and surveys featuring Likert-scale and open-ended questions. Quantitative analysis of key learning outcomes revealed significant improvements in comprehension and retention, evidenced by enhanced pre-and post- test scores (p < 0.01 and p < 0,05, respectively) and motivation, as indicated by increased engagement in survey responses (p < 0.05). Qualitative analysis revealed a need for more examples and visual aids to complement analogies and a preference for balancing analogies with detailed technical content. This study demonstrates the potential of Al-generated analogies to make complex Al concepts more accessible and engaging. Future research should refine analogy generation. incorporate multimedia elements, and explore long-term and cross-cultural impacts to further enhance Al education.

Artificial intelligence (AI) applications to support human tutoring have potential to significantly improve learning outcomes, but engagement issues persist, especially among students from low-income backgrounds. We introduce an AI-assisted tutoring model that combines human and AI tutoring and hypothesize this synergy will have positive impacts on learning processes. To investigate this hypothesis, we conduct a three-study quasi-experiment across three urban and low-income middle schools: 1) 125 students in a Pennsylvania school; 2) 385 students (50% Latinx) in a California school, and 3) 75 students (100% Black) in a Pennsylvania charter school, all implementing analogous tutoring models. We compare learning analytics of students engaged in human-AI tutoring compared to students using math software only. We find human-AI tutoring has positive effects, particularly in student’s proficiency and usage, with evidence suggesting lower achieving students may benefit more compared to higher achieving students. We illustrate the use of quasi-experimental methods adapted to the particulars of different schools and data-availability contexts so as to achieve the rapid data-driven iteration needed to guide an inspired creation into effective innovation. Future work focuses on improving the tutor dashboard and optimizing tutor-student ratios, while maintaining annual costs per student of approximately $700 annually.

We developed GPTutor, a pioneering web application designed to revolutionize personalized learning by leveraging the capabilities of Generative AI at scale. GPTutor adapts educational content and practice exercises to align with individual students' interests and career goals, enhancing their engagement and understanding of critical academic concepts. The system uses a serverless architecture to deliver personalized and scalable learning experiences. By integrating advanced Chain-of-Thoughts prompting methods, GPTutor provides a personalized educational journey that not only addresses the unique interests of each student but also prepares them for future professional success. This demo paper presents the design, functionality, and potential of GPTutor to foster a more engaging and effective educational environment.

Written feedback has long been a cornerstone in educational and professional settings, essential for enhancing learning outcomes. However, multimodal feedback-integrating textual, auditory, and visual cues-promises a more engaging and effective learning experience. By leveraging multiple sensory channels, multimodal feedback better accommodates diverse learning preferences and aids in deeper information retention. Despite its potential, creating multimodal feedback poses challenges, including the need for increased time and resources. Recent advancements in generative artificial intelligence (GenAI) offer solutions to automate the feedback process, predominantly focusing on textual feedback. Yet, the application of GenAI in generating multimodal feedback remains largely unexplored. Our study investigates the use of GenAI techniques to generate multimodal feedback, aiming to provide this feedback for large cohorts of learners, thereby enhancing learning experience and engagement. By exploring the potential of GenAI for this purpose, we propose a framework for automating the generation of multimodal feedback, which we name MuFIN.

Automated feedback systems are pivotal for scaling personalized learning, especially when dealing with large cohorts of learners.This paper introduces HAROR (Highlighting and Rephrasing Openended Responses), a feedback system that utilizes the advanced capabilities of Generative Pre-trained Transformer (GPT) models, including GPT-4 and GPT-3.5, to provide explanatory feedback on learner responses (trainee tutors as learners in our study) to openended questions.HAROR can identify desirable and undesirable parts of open-ended responses, offer explanatory feedback, and rephrase the undesired responses into desirable forms, aiming to foster learners' understanding and improvement.

Addressing negative self-talk by students, such as responding to a student when saying, "I am dumb"or "I can't do this"can be difficult for even the most experienced tutor. Despite potential tutor learning from scenario-based lessons on this topic, human-graded assessment remains time-consuming. Leveraging generative AI for evaluating textual responses in online training presents a scalable solution. Research suggests a tutor validates student's feelings when they speak negatively of themselves, e.g., by a tutor responding, "I understand how you feel"or "I recognize this is difficult."This ongoing work assesses the performance of 60 undergraduate tutors within an online lesson on enhancing tutors' abilities to respond to students engaging in negative self-talk. We find statistically significant tutor learning gains from pretest to posttest. Additionally, we describe a method of using generative AI for assessing tutors' responses to predict the best approach and subsequently explain the rationale behind it. Using the large language model GPT-4, we find high absolute performance when evaluating tutor responses involving predicting (F1 = 0.85) and explaining (F1 = 0.83) the best approach. Minor improvements are needed to the lesson itself. A future goal of this work is to fully develop automated systems of assessing tutor learning attending to barriers to students' motivation and doing so at scale.

Educational feedback has been widely acknowledged as an effective approach to improving student learning. However, scaling effective practices can be laborious and costly, which motivated researchers to work on automated feedback systems (AFS). Inspired by the recent advancements in the pre-trained language models (e.g., ChatGPT), we posit that such models might advance the existing knowledge of textual feedback generation in AFS because of their capability to offer natural-sounding and detailed responses. Therefore, we aimed to investigate the feasibility of using ChatGPT to provide students with feedback to help them learn better. Our results show that i) ChatGPT is capable of generating more detailed feedback that fluently and coherently summarizes students' performance than human instructors; ii) ChatGPT achieved high agreement with the instructor when assessing the topic of students' assignments; and iii) ChatGPT could provide feedback on the process of students completing the task, which might benefit students developing learning skills.

Feedback is an effective way to assist students in achieving learning goals. The conceptualisation of feedback is gradually moving from feedback as information to feedback as a learner-centred process. To demonstrate feedback effectiveness, feedback as a learner-centred process should be designed to provide quality feedback content and promote student learning outcomes on the subsequent task. However, it remains unclear how instructors adopt the learner-centred feedback framework for feedback provision in the teaching practice. Thus, our study made use of a comprehensive learner-centred feedback framework to analyse feedback content and identify the characteristics of feedback content among student groups with different performance changes. Specifically, we collected the instructors’ feedback on two consecutive assignments offered by an introductory to data science course at the postgraduate level. On the basis of the first assignment, we used the status of student grade changes (i.e., students whose performance increased and those whose performance did not increase on the second assignment) as the proxy of the student learning outcomes. Then, we engineered and extracted features from the feedback content on the first assignment using a learner-centred feedback framework and further examined the differences of these features between different groups of student learning outcomes. Lastly, we used the features to predict student learning outcomes by using widely-used machine learning models and provided the interpretation of predicted results by using the SHapley Additive exPlanations (SHAP) framework. We found that 1) most features from the feedback content presented significant differences between the groups of student learning outcomes, 2) the gradient boost tree model could effectively predict student learning outcomes, and 3) SHAP could transparently interpret the feature importance on predictions.

Existing research indicates that students prefer to work with tutors who express politely in online human-human tutoring, but excessive polite expressions might lower tutoring efficacy. However, there is a shortage of understanding about the use of politeness in online tutoring and the extent to which the politeness of instructional strategies can contribute to students’ achievement. To address these gaps, we conducted a study on a large-scale dataset (5,165 students and 116 qualified tutors in 18,203 online tutoring sessions) of both effective and ineffective human-human online tutorial dialogues. The study made use of a well-known dialogue act coding scheme to identify instructional strategies, relied on the linguistic politeness theory to analyse the politeness levels of the tutors’ instructional strategies, and utilised Gradient Tree Boosting to evaluate the predictive power of these politeness levels in revealing students’ problem-solving performance. The results demonstrated that human tutors used both polite and non-polite expressions in the instructional strategies. Tutors were inclined to express politely in the strategy of providing positive feedback but less politely while providing negative feedback and asking questions to evaluate students’ understanding. Compared to the students with prior progress, tutors provided more polite open questions to the students without prior progress but less polite corrective feedback. Importantly, we showed that, compared to previous research, the accuracy of predicting student problem-solving performance can be improved by incorporating politeness levels of instructional strategies with other documented predictors (e.g., the sentiment of the utterances).

The HCI community has a rich history of finding new ways to engage people with data beyond the screen. With our work, we aim to expand the scope of how interaction design can engage people, arguing that “eating data” has the potential to allow people to experience “data as delight”. With reference to prior work and our design research findings, we discuss the advantages and the challenges of this approach to integrating data and food. We then identify four themes to guide the design of engagements with data through food: food form, food commensality, food ephemerality, and emotional response to food. Within these design themes, we articulate twelve insights for interaction designers to use when working on serving data as delight.

This research investigates whether students’ level of domain expertise can be detected during authentic learning activities by analyzing their physical activity patterns. More expert students reduced their manual activity by a substantial 50%, which was evident in fine-grained signal analyses and total rate of gesturing. The quality of experts’ discrete hand movements also averaged shorter in distance, briefer in duration, and slower in velocity than those of non-experts. Interestingly, experts adapted by nearly eliminating gestures on easier problems, while selectively increasing them on harder ones. They also strategically produced 62% more iconic gestures, which serve to retain spatial information in working memory while extracting inferences required to solve problems correctly. These findings highlight the close relation between hand movements and mental state and, more specifically, that hand movements provide an unusually clear window on students’ level of domain expertise. Embodied Cognition and Limited Resource theories only partially account for the present findings, which specify future directions for theoretical work.

Affective computing is an emerging research area which provides insights on human's mental state through human-machine interaction. During the interaction process, bio-signal analysis is essential to detect human affective changes. Currently, machine learning methods to analyse bio-signals are the state of the art to detect the affective states, but most empirical works mainly deploy traditional machine learning methods rather than deep learning models due to the need for explainability. In this paper, we propose a deep learning model to process multimodal-multisensory bio-signals for affect recognition. It supports batch training for different sampling rate signals at the same time, and our results show significant improvement compared to the state of the art. Furthermore, the results are interpreted at the sensor- and signal- level to improve the explainaibility of our deep learning model.

Embodied Cognition theorists believe that mathematics thinking is embodied in physical activity, like gesturing while explaining math solutions. This research asks the question whether expertise in mathematics can be detected by analyzing students’ rate and type of manual gestures. The results reveal several unique findings, including that math experts reduced their total rate of gesturing by 50%, compared with non-experts. They also dynamically increased their rate of gesturing on harder problems. Although experts reduced their rate of gesturing overall, they selectively produced 62% more iconic gestures. Iconic gestures are strategic because they assist with retaining spatial information in working memory, so that inferences can be extracted to support correct problem solving. The present results on representation-level gesture patterns are convergent with recent findings on signal-level handwriting, while also contributing a causal understanding of how and why experts adapt their manual activity during problem solving.