With the emergence of generative AI, the field of Learning Analytics (LA) has increasingly embraced the use of Large Language Models (LLMs) to automate qualitative analysis. Deductive analysis requires theoretical or other conceptual grounding to inform coding. However, few studies detail the process of translating the literature into a codebook, and then into an effective LLM prompt. In this paper, we introduce Grounded Prompt Engineering (GROPROE) as a systematic process to develop a literature-grounded prompt for deductive analysis. We demonstrate our GROPROE process on a dataset of 860 written reflections, coding for students’ affective engagement and sense of belonging. To evaluate the quality of the coding we demonstrate substantial human/LLM Inter-Annotator Reliability (IAR). To evaluate the consistency of LLM coding, a subset of the data was analysed 60 times using the LLM Quotient showing how this stabilized for most codes. We discuss the dynamics of human-AI interaction when following GROPROE, foregrounding how the prompt took over as the iteratively revised codebook, and how the LLM provoked codebook revision. The contributions to the LA field are threefold: (i) GROPROE as a systematic prompt-design process for deductive coding grounded in literature, (ii) a detailed worked example showing its application to Belonging Analytics, and (iii) implications for human-AI interaction in automated deductive analysis.

A student's “sense of belonging” is critical to retention and success in higher education. However, belonging is a multifaceted and dynamic concept, making monitoring and supporting it with timely action challenging. Conventional approaches to researching belonging depend on lengthy surveys and/or focus groups, and while often insightful, these are resource-intensive, slow, and cannot be repeated too often. “Belonging Analytics” is an emerging concept pointing to the potential of learning analytics to address this challenge, and to illustrate this concept, this paper investigates the feasibility of asking students a single question about what promotes their sense of belonging. To validate this, responses were analysed using a form of topic modelling, and these were triangulated by examining alignment with (i) students’ responses to Likert scale items in a belonging scale and (ii) the literature on the drivers of belonging. These alignments support our proposal that this is a practical tool to gain timely insight into a cohort's sense of belonging. Reflecting our focus on practical tools, the approach is implemented using analytics products readily available to educational institutions — Linguistic Inquiry Word Count (LIWC) and Statistical Program for Social Sciences (SPSS).

Feedback is an effective way to assist students in achieving learning goals. The conceptualisation of feedback is gradually moving from feedback as information to feedback as a learner-centred process. To demonstrate feedback effectiveness, feedback as a learner-centred process should be designed to provide quality feedback content and promote student learning outcomes on the subsequent task. However, it remains unclear how instructors adopt the learner-centred feedback framework for feedback provision in the teaching practice. Thus, our study made use of a comprehensive learner-centred feedback framework to analyse feedback content and identify the characteristics of feedback content among student groups with different performance changes. Specifically, we collected the instructors’ feedback on two consecutive assignments offered by an introductory to data science course at the postgraduate level. On the basis of the first assignment, we used the status of student grade changes (i.e., students whose performance increased and those whose performance did not increase on the second assignment) as the proxy of the student learning outcomes. Then, we engineered and extracted features from the feedback content on the first assignment using a learner-centred feedback framework and further examined the differences of these features between different groups of student learning outcomes. Lastly, we used the features to predict student learning outcomes by using widely-used machine learning models and provided the interpretation of predicted results by using the SHapley Additive exPlanations (SHAP) framework. We found that 1) most features from the feedback content presented significant differences between the groups of student learning outcomes, 2) the gradient boost tree model could effectively predict student learning outcomes, and 3) SHAP could transparently interpret the feature importance on predictions.

Email communication between instructors and students is ubiquitous, and it could be valuable to explore ways of testing out how to make email messages more impactful. This paper explores the design space of using emails to get students to plan and reflect on starting weekly homework earlier. We deployed a series of email reminders using randomized A/B comparisons to test alternative factors in the design of these emails, providing examples of an experimental paradigm and metrics for a broader range of interventions. We also surveyed and interviewed instructors and students to compare their predictions about the effectiveness of the reminders with their actual impact. We present our results on which seemingly obvious predictions about effective emails are not borne out, despite there being evidence for further exploring these interventions, as they can sometimes motivate students to attempt their homework more often. We also present qualitative evidence about student opinions and behaviours after receiving the emails, to guide further interventions. These findings provide insight into how to use randomized A/B comparisons in everyday channels such as emails, to provide empirical evidence to test our beliefs about the effectiveness of alternative design choices.

Academic analytics focuses on collecting, analysing and visualising educational data to generate institutional insights and improve decision-making for academic purposes. However, challenges that arise from navigating a complex organisational structure when introducing analytics systems have called for the need to engage key stakeholders widely to cultivate a shared vision and ensure that implemented systems create desired value. This paper presents a study that takes co-design steps to identify design needs and strategic approaches for the adoption of academic analytics, which serves the purpose of enhancing the measurement of educational quality utilising institutional data. Through semi-structured interviews with 54 educational stakeholders at a large research university, we identified particular interest in measuring student engagement and the performance of courses and programmes. Based on the observed perceptions and concerns regarding data use to measure or evaluate these areas, implications for adoption strategy of academic analytics, such as leadership involvement, communication, and training, are discussed.

Learning analytics (LA) has been presented as a viable solution for scaling timely and personalised feedback to support students’ self-regulated learning (SRL). Research is emerging that shows some positive associations between personalised feedback with students’ learning tactics and strategies as well as time management strategies, both important aspects of SRL. However, the definitive role of feedback on students’ SRL adaptations is under-researched; this requires an examination of students’ recalled experiences with their personalised feedback. Furthermore, an important consideration in feedback impact is the course context, comprised of the learning design and delivery modality. This mixed-methods study triangulates learner trace data from two different course contexts, with students’ qualitative data collected from focus group discussions, to more fully understand the impact of their personalised feedback and to explicate the role of this feedback on students’ SRL adaptations. The quantitative analysis showed the contextualised impact of the feedback on students’ learning and time management strategies in the different courses, while the qualitative analysis highlighted specific ways in which students used their feedback to adjust these and other SRL processes.

This paper reports on the findings of a study that proposed a novel learning analytics methodology that combines three complimentary techniques - agglomerative hierarchical clustering, epistemic network analysis, and process mining. The methodology allows for identification and interpretation of self-regulated learning in terms of the use of learning strategies. The main advantage of the new technique over the existing ones is that it combines the time management and learning tactic dimensions of learning strategies, which are typically studied in isolation. The new technique allows for novel insights into learning strategies by studying the frequency of, strength of connections between, and ordering and time of execution of time management and learning tactics. The technique was validated in a study that was conducted on the trace data of first-year undergraduate students who were enrolled into two consecutive offerings (N2017 = 250 and N2018 = 232) of a course at an Australian university. The application of the proposed technique identified four strategy groups derived from three distinct time management tactics and five learning tactics. The tactics and strategies identified with the technique were correlated with academic performance and were interpreted according to the established theories and practices of self-regulated learning.

Learning Analytics Dashboards (LAD) are becoming an increasingly popular way to provide students with personalised feedback. Despite the number of LADs being developed, significant research gaps exist around the student perspective, especially how students make sense of graphics provided in LADs, and how they intend to act on the feedback provided therein. This study employed a randomized-controlled trial to examine students' sense-making of LADs showing four different frames of reference, and to what extent the impact of LADs was mediated by baseline self-regulation. Using a mix of quantitative and qualitative data analysis, the results revealed rather distinct patterns in students' sense-making across the four LADs. These patterns involved the intersection of visual salience and planned learning actions. However, collectively, across all four LADs a consistent theme emerged around students planned learning actions. This theme was classified as time and study environment management. A key finding of the study is that the use of LADs as a primary feedback process should be personalized and include training and support to aid student sensemaking.

Video materials have become an integral part of university learning and teaching practice. While empirical research concerning the use of videos for educational purposes has increased, the literature lacks an overview of the specific effects of videos on diverse learning outcomes. To address such a gap, this paper presents preliminary results of a large-scale systematic review of peer-reviewed empirical studies published from 2007-2017. The study synthesizes the trends observed through the analysis of 178 papers selected from the screening of 2531 abstracts. The findings summarize the effects of manipulating video presentation, content and tasks on learning outcomes, such as recall, transfer, academic achievement, among others. The study points out the gap between large-scale analysis of fine-grained data on video interaction and experimental findings reliant on established psychological instruments. Narrowing this gap is suggested as the future direction for the research on video-based learning.