This position paper builds on previous research publications and activities related to trustworthy learning analytics (LA) to provide an additional angle on the fundamental considerations for ensuring trustworthy LA. In our view, these considerations include strategic guidance and support, pedagogical soundness and human interaction, stakeholder engagement, data and AI literacy, ethics, data limitations and meaningful use of algorithms, as well as transparency of the whole process. In this paper, we discuss each of the considerations with respect to the roles and responsibilities of the key stakeholders in the LA systems: educational leaders, educators (especially teachers) and students.

Substantial progress has been made in how educators can be supported to implement effective learning design (LD) with learning analytics (LA). However, how educators make micro-decisions about designing individual teaching and learning activities (TLAs) and how these are related to wider pedagogical approaches has received limited empirical support. This study explored how 165 educators designed and integrated 12,749 TLA in 218 LDs using clustering, pattern-mining, and correlational analysis. The findings suggest most educators use a combination of four common LD TLAs (i.e., Collaboration, Generating independent learning, Assessment, and Traditional classroom activities). The four common TLAs could be used to develop LA and Generative Artificial Intelligence (Gen-AI) approaches to support educators in making more informed and evidence-based design decisions for effective learning and teaching.

Generative artificial intelligence (GenAI) tools, such as large language models (LLMs), generate natural language and other types of content to perform a wide range of tasks. This represents a significant technological advancement that poses opportunities and challenges to educational research and practice. This commentary brings together contributions from nine experts working in the intersection of learning and technology and presents critical reflections on the opportunities, challenges, and implications related to GenAI technologies in the context of education. In the commentary, it is acknowledged that GenAI’s capabilities can enhance some teaching and learning practices, such as learning design, regulation of learning, automated content, feedback, and assessment. Nevertheless, we also highlight its limitations, potential disruptions, ethical consequences, and potential misuses. The identified avenues for further research include the development of new insights into the roles human experts can play, strong and continuous evidence, human-centric design of technology, necessary policy, and support and competence mechanisms. Overall, we concur with the general skeptical optimism about the use of GenAI tools such as LLMs in education. Moreover, we highlight the danger of hastily adopting GenAI tools in education without deep consideration of the efficacy, ecosystem-level implications, ethics, and pedagogical soundness of such practices.

Learning Analytics Dashboards (LAD) are the subject of research in a multitude of schools and higher education institutions, but a lack of research into learner-facing dashboards in professional learning has been identified. This study took place in an authentic professional learning context and aims to contribute insights into LAD design by using an academic approach in a practice-based environment. An existing storytelling LAD created to support 81 accountants was evaluated using Technology Acceptance Model, finding a learner expectation for clarity, conciseness, understanding and guidance on next steps. High usage levels and a ‘take what you need’ approach was identified, with all visualizations and automated personalized feedback being considered useful although to varying degrees. Professional learners in this study focus on understanding and acting upon weaknesses rather than celebrating strengths. The lessons for LAD design are to offer choice and create elements which support learners to take action to improve performance at a multitude of time points and levels of success.

Accurate and timely measurement of learning engagement is crucial for the application of precision education. At the same time, it is still a central research theme, both in the learning analytics community as in the broader area of educational research. 'Engagement is one of the hottest research topics in the field of educational psychology' is for a good reason the opening sentence of a recent special issue. In our contribution, we propose a holistic approach to the measurement of engagement by integrating data of behavioral type through traces of learning processes captured from log files into affective, behavioral, and cognitive measures of engagement collected with surveys and cognitive measures from assessments for and as learning. We apply this holistic approach in an empirical analysis of dispositional learning analytics. Starting from four different engagement profiles created by two-step clustering, we find that these profiles primarily differ in their timing of engagement with learning. Next, we develop regression-based prediction models that make clear that trace, survey, and assessment data have complementary roles in signaling students at risk for failure and are all three crucial constituents of prediction equations that differ in the timing of learning feedback.

The identification of students’ learning strategies by using multi-modal data that combine trace data with self-report data is the prime aim of this study. Our context is an application of dispositional learning analytics in a large introductory course mathematics and statistics, based on blended learning. Building on previous studies in which we found marked differences in how students use worked examples as a learning strategy, we compare different profiles of learning strategies on learning dispositions and learning outcome. Our results cast a new light on the issue of efficiency of learning by worked examples, tutored and untutored problem-solving: in contexts where students can apply their own preferred learning strategy, we find that learning strategies depend on learning dispositions. As a result, learning dispositions will have a confounding effect when studying the efficiency of worked examples as a learning strategy in an ecologically valid context.

The identification of students’ learning strategies by using multi-modal data that combine trace data with self-report data is the prime aim of this study. Our context is an application of dispositional learning analytics in a large introductory course mathematics and statistics, based on blended learning. Building on previous studies in which we found marked differences in how students use worked examples as a learning strategy, we compare different profiles of learning strategies on learning dispositions and learning outcome. Our results cast a new light on the issue of efficiency of learning by worked examples, tutored and untutored problem-solving: in contexts where students can apply their own preferred learning strategy, we find that learning strategies depend on learning dispositions. As a result, learning dispositions will have a confounding effect when studying the efficiency of worked examples as a learning strategy in an ecologically valid context.

In recent years, the connection between Learning Design (LD) and Learning Analytics (LA) has been emphasized by many scholars as it could enhance our interpretation of LA findings and translate them to meaningful interventions. Together with numerous conceptual studies, a gradual accumulation of empirical evidence has indicated a strong connection between how instructors design for learning and student behaviour. Nonetheless, students' timing of engagement and its relation to LD and academic performance have received limited attention. Therefore, this study investigates to what extent students' timing of engagement aligned with instructor learning design, and how engagement varied across different levels of performance. The analysis was conducted over 28 weeks using trace data, on 387 students, and replicated over two semesters in 2015 and 2016. Our findings revealed a mismatch between how instructors designed for learning and how students studied in reality. In most weeks, students spent less time studying the assigned materials on the VLE compared to the number of hours recommended by instructors. The timing of engagement also varied, from in advance to catching up patterns. High-performing students spent more time studying in advance, while low-performing students spent a higher proportion of their time on catching-up activities. This study reinforced the importance of pedagogical context to transform analytics into actionable insights.

This study aims to contribute to recent developments in empirical studies on students' learning strategies, whereby the use of trace data is combined with self-report data to distinguish profiles of learning strategy use [3--5]. We do so in the context of an application of dispositional learning analytics in a large introductory course mathematics and statistics, based on blended learning. Building on our previous work which showed marked differences in how students used worked examples as a learning strategy [7, 11], this study compares different profiles of learning strategies with learning approaches, learning outcomes, and learning dispositions. One of our key findings is that deep learners were less dependent on worked examples as a resource for learning, and that students who only sporadically used worked examples achieved higher test scores.

In this paper, we describe a large-scale study about the use of predictive learning analytics data with 240 teachers in 10 modules at a distance learning higher education institution. The aim of the study was to illuminate teachers' uses and practices of predictive data, in particular identify how predictive data was used to support students at risk of not completing or failing a module. Data were collected from statistical analysis of 17,033 students' performance by the end of the intervention, teacher usage statistics, and five individual semi-structured interviews with teachers. Findings revealed that teachers endorse the use of predictive data to support their practice yet in diverse ways and raised the need for devising appropriate intervention strategies to support students at risk.

Substantial progress has been made in understanding how teachers design for learning. However, there remains a paucity of evidence of the actual students' response towards leaning designs. Learning analytics has the power to provide just-in-time support, especially when predictive analytics is married with the way teachers have designed their course, or so-called a learning design. This study investigates how learning designs are configured over time and their impact on student activities by analyzing longitudinal data of 38 modules with a total of 43,099 registered students over 30 weeks at the Open University UK, using social network analysis and panel data analysis. Our analysis unpacked dynamic configurations of learning designs between modules over time, which allows teachers to reflect on their practice in order to anticipate problems and make informed interventions. Furthermore, by controlling for the heterogeneity between modules, our results indicated that learning designs were able to explain up to 60% of the variability in student online activities, which reinforced the importance of pedagogical context in learning analytics.

This study provides a conceptual framework how organizations may adopt evidence-based interventions at scale, and how institutions may evaluate the costs and benefits of such interventions. Building on a new conceptual model developed by the Open University UK (OU), we will analyse three case-studies of evidence-based interventions. By working with 90+ large-scale modules for a period of two years across the five faculties and disciplines within the OU, Analytics4Action provides a bottom-up-approach for working together with key stakeholders within their respective contexts. Using principles of embedded case-study approaches by Yin [1], by comparing the learning behavior, satisfaction and performance of 11079 learners the findings indicated that each of the three learning designs led to satisfied students and average to good student retention. In the second part we highlighted that the three module teams made in-presentation interventions based upon real-time analytics, whereby initial user data indicated VLE behaviour in line with expectations. In 2-5 years, we hope that a rich, robust evidence-base will be presented to show how learning analytics can help teachers to make informed, timely and successful interventions that will help learners to achieve their learning outcomes.

An increasing number of researchers are taking learning design into consideration when predicting learning behavior and outcomes across different modules. This study builds on preliminary learning design work that was presented at LAK2015 by the Open University UK. In this study we linked 151 modules and 111.256 students with students' satisfaction and performance using multiple regression models. Our findings strongly indicate the importance of learning design in predicting and understanding performance of students in blended and online environments. In line with proponents of social learning analytics, our primary predictor for academic retention was the amount of communication activities, controlling for various institutional and disciplinary factors. Where possible, appropriate communication tasks that align with the learning objectives of the course may be a way forward to enhance academic retention.

Learning analytics seek to enhance the learning processes through systematic measurements of learning related data and to provide informative feedback to learners and educators. Track data from Learning Management Systems (LMS) constitute a main data source for learning analytics. This empirical contribution provides an application of Buckingham Shum and Deakin Crick’s theoretical framework of dispositional learning analytics: an infrastructure that combines learning dispositions data with data extracted from computer-assisted, formative assessments and LMSs. In two cohorts of a large introductory quantitative methods module, 2049 students were enrolled in a module based on principles of blended learning, combining face-to-face Problem-Based Learning sessions with e-tutorials. We investigated the predictive power of learning dispositions, outcomes of continuous formative assessments and other system generated data in modelling student performance and their potential to generate informative feedback. Using a dynamic, longitudinal perspective, computer-assisted formative assessments seem to be the best predictor for detecting underperforming students and academic performance, while basic LMS data did not substantially predict learning. If timely feedback is crucial, both use-intensity related track data from e-tutorial systems, and learning dispositions, are valuable sources for feedback generation.

While substantial progress has been made in terms of predictive modeling in the Learning Analytics Knowledge (LAK) community, one element that is often ignored is the role of learning design. Learning design establishes the objectives and pedagogical plans which can be evaluated against the outcomes captured through learning analytics. However, no empirical study is available linking learning designs of a substantial number of courses with usage of Learning Management Systems (LMS) and learning performance. Using cluster- and correlation analyses, in this study we compared how 87 modules were designed, and how this impacted (static and dynamic) LMS behavior and learning performance. Our findings indicate that academics seem to design modules with an "invisible" blueprint in their mind. Our cluster analyses yielded four distinctive learning design patterns: constructivist, assessment-driven, balanced-variety and social constructivist modules. More importantly, learning design activities strongly influenced how students were engaging online. Finally, learning design activities seem to have an impact on learning performance, in particular when modules rely on assimilative activities. Our findings indicate that learning analytics researchers need to be aware of the impact of learning design on LMS data over time, and subsequent academic performance.