The recent advances in educational technology enabled the development of solutions that collect and analyse data from learning scenarios to inform the decision-making processes. Research fields like Learning Analytics (LA) and Artificial Intelligence (AI) aim at supporting teaching and learning by using such solutions. However, their adoption in authentic settings is still limited, among other reasons, derived from ignoring the stakeholders' needs, a lack of pedagogical contextualisation, and a low trust in new technologies. Thus, the research fields of Human-Centered LA (HCLA) and Human-Centered AI (HCAI) recently emerged, aiming to understand the active involvement of stakeholders in the creation of such proposals. This paper presents a systematic literature review of 47 empirical research studies on the topic. The results show that more than two-thirds of the papers involve stakeholders in the design of the solutions, while fewer papers involved them during the ideation and prototyping, and the majority do not report any evaluation. Interestingly, while multiple techniques were used to collect data (mainly interviews, focus groups and workshops), few papers explicitly mentioned the adoption of existing HC design guidelines. Further evidence is needed to show the real impact of HCLA/HCAI approaches (e.g., in terms of user satisfaction and adoption).

Multimodal learning analytics (MMLA) research for building collaboration quality estimation models has shown significant progress. However, the generalizability of such models is seldom addressed. In this paper, we address this gap by systematically evaluating the across-context generalizability of collaboration quality models developed using a typical MMLA pipeline. This paper further presents a methodology to explore modelling pipelines with different configurations to improve the generalizability of the model. We collected 11 multimodal datasets (audio and log data) from face-to-face collaborative learning activities in six different classrooms with five different subject teachers. Our results showed that the models developed using the often-employed MMLA pipeline degraded in terms of Kappa from Fair (.20 < Kappa < .40) to Poor (Kappa < .20) when evaluated across contexts. This degradation in performance was significantly ameliorated with pipelines that emerged as high-performing from our exploration of 32 pipelines. Furthermore, our exploration of pipelines provided statistical evidence that often-overlooked contextual data features improve the generalizability of a collaboration quality model. With these findings, we make recommendations for the modelling pipeline which can potentially help other researchers in achieving better generalizability in their collaboration quality estimation models.

Multimodal Learning Analytics (MMLA) has been applied to collaborative learning, often to estimate collaboration quality with the use of multimodal data, which often have uneven time scales. The difference in time scales is usually handled by dividing and aggregating data using a fixed-size time window. So far, the current MMLA research lacks a systematic exploration of whether and how much window size affects the generalizability of collaboration quality estimation models. In this paper, we investigate the impact of different window sizes (e.g., 30 seconds, 60s, 90s, 120s, 180s, 240s) on the generalizability of classification models for collaboration quality and its underlying dimensions (e.g., argumentation). Our results from an MMLA study involving the use of audio and log data showed that a 60 seconds window size enabled the development of more generalizable models for collaboration quality (AUC 61%) and argumentation (AUC 64%). In contrast, for modeling dimensions focusing on coordination, interpersonal relationship, and joint information processing, a window size of 180 seconds led to better performance in terms of across-context generalizability (on average from 56% AUC to 63% AUC). These findings have implications for the eventual application of MMLA in authentic practice.

Over the past decade, the use of chatbots for educational purposes has gained considerable traction. A similar trend has been observed in social coding platforms, where automated agents support software developers with tasks such as performing code reviews. While incorporating code reviews and social coding platforms into software engineering education has been found to be beneficial, challenges such as steep learning curves and privacy considerations are barriers to their adoption. Furthermore, no study has addressed the role chatbots play in supporting code reviews as a pedagogical tool. To help address this gap, we developed an online learning application that simulates the code review features available on social coding platforms and allows instructors to interact with students using chatbot identities. We then embedded this application within a lesson on software engineering best practices and conducted a controlled in-class experiment. This experiment examined the effect that explaining content via chatbot identities had on three aspects: (i) students’ perceived usability of the lesson, (ii) their engagement with the code review process, and (iii) their learning gains. While our findings show that it is feasible to simulate the code review process within an online learning platform and achieve good usability, our quantitative analysis did not yield significant differences across treatment conditions for any of the aspects considered. Nevertheless, our qualitative results suggest that students expect explicit feedback when performing this type of exercise and could thus benefit from automated replies provided by an interactive chatbot. We propose to build on our current findings to further explore this line of research in future work.

The trend towards generalized blended and active learning approaches in engineering education is redefining the requirements and the deployment schemes for digital education platforms. This paper presents a quadriptych techno-pedagogical model highlighting the features and the services an effective digital education platform should provide to accompany educators in their journey towards the creation of rich—and possibly open—educational resources, as well as their integration into blended and active learning scenarios. An implementation of this model for a novel digital education platform called Graasp is discussed and three use cases are detailed. These use cases show that a large variety of active pedagogical scenarios can be implemented, if easily accessible and comprehensive digital education platforms are available.

Peer code review has proven to be a valuable tool in software engineering. However, integrating code reviews into educational contexts is particularly challenging due to the complexity of both the process and popular code review tools. We propose to address this challenge by designing a code review application (CRA) aimed at teaching the code review process directly within existing online learning platforms. Using the CRA, instructors can scaffold online lessons that introduce the code review process to students through code snippets, following a format resembling computational notebooks. We refer to this online lesson format as the code review notebook format. Through a case study comprising an online lesson on code quality standards completed by 23 university students, we evaluated the usability of the CRA and the code review notebook format, obtaining positive results for both. These results are a first step toward integrating code review notebooks into software engineering education.

Multimodal Learning Analytics (MMLA) aims to support evidence-based educational practices by collecting, processing, analyzing and sense-making of multimodal evidence of learning. MMLA is not widespread yet and there are few tailored MMLA solutions to meet the requirements of a specific learning scenario. This PhD project investigates the main challenges behind the limited development of MMLA solutions and proposes an MMLA infrastructure as the main contribution. The proposed infrastructure includes three components - a data value chain, a data model and a software architecture. The overall project follows the design-based research methodology where multiple iterations are involved to refine the contributions.

Despite the traction gained by the open data movement and the rise of big data and learning analytics in education, there is limited support for researchers in education to generate, access, and share experimental data using openly-available digital education platforms. To explore how this gap could be addressed and elicit requirements, we conducted a survey with 40 researchers in the field of technology-enhanced learning, examining their experience and needs handling research data. Drawing on the results of our survey, we devised a set of features that educational platforms should provide to address the identified requirements, enabling researchers in education to run studies within typical learning environments, adhere to legal and ethical frameworks concerning privacy, and share their data confidently with a wider audience. We then categorized these features into five stages that represent the user flow, namely (1) Bootstrapping Research Studies, (2) Ensuring Consent, (3) Gathering Data, (4) Managing Data Sets, and (5) Supporting Open Research and Collaboration. Our aim is to guide forthcoming research and developments to relieve researchers of the burdens of conducting data-sensitive experiments, support the adoption of best practices, and pave the way for open data policies in digital education.

The need to ensure privacy and data protection in educational contexts is driving a shift towards new ways of securing and managing learning records. Although there are platforms available to store educational activity traces outside of a central repository, no solution currently guarantees that these traces are authentic when they are retrieved for review. This paper presents a blueprint for an architecture that employs blockchain technology to sign and validate learning traces, allowing them to be stored in a distributed network of repositories without diminishing their authenticity. Our proposal puts participants in online learning activities at the center of the design process, granting them the option to store learning traces in a location of their choice. Using smart contracts, stakeholders can retrieve the data, securely share it with third parties and ensure it has not been tampered with, providing a more transparent and reliable source for learning analytics. Nonetheless, a preliminary evaluation found that only 56% of teachers surveyed considered tamper-evident storage a useful feature of a learning trace repository. These results motivate further examination with other end users, such as learning analytics researchers, who may have stricter expectations of authenticity for data used in their practice.

Collaborative learning flow patterns (CLFPs) encode solutions to recurrent pedagogical problems, which have been successfully applied to the design of learning experiences. However, the pedagogical knowledge encoded in these patterns has seldom been exploited in learning analytics (LA). This paper analyzes four of the most common CLFPs to extract the intrinsic constraints that lead to a successful collaborative learning activity, and use them to enhance existing LA solutions. To understand the added value of applying such codified knowledge in LA, we present evidence from five authentic case studies in which such constraints aided university teachers in monitoring complex collaborative scripts. The results not only illustrate quantitatively such added value but also unearth qualitative benefits, such as raising practitioners' awareness about how the current state of activities may affect future phases of the script.

The collection of evidence from authentic classroom practice is important both for teacher professional development (TPD) and educational research. However, most current learning analytics (LA) fails to capture the physical occurrences of the classroom, and do not always address individual teachers' needs. More customizable forms of classroom data collection (e.g., through video-recordings or by human observers) are time-consuming and expensive to implement. Navigating this tradeoff between the personalization of data collection and the strict time/effort constraints of classroom practice is still an unsolved challenge for designers of LA and teaching analytics (TA) systems to be used in face-to-face classrooms. In this paper, we extract lessons learnt from a design-based research process that explores this tradeoff, towards the development of teacher-driven, personalizable data collection tools. Through a survey of 15 expert teachers and paper and software prototype testing with a total of 14 teachers in simulated and authentic settings, we gathered information about teachers' preferences for teacher-driven data collection in classrooms, and derived design insights (e.g., a cold-start problem and the need for multiple layers in the personalization), which can be useful for the design of TA/LA tools that collect personalized data from everyday classrooms.

There is an increasing interest in Multimodal Learning Analytics (MMLA), which involves complex technical issues in gathering, merging and analyzing different types of learning data from heterogeneous data sources. However, there is still no common reference architecture to face these technical challenges of MMLA. This paper summarizes the state of the art of MMLA software architectures through a systematic literature review. Our analysis of nine architecture proposals highlights the uneven support provided by existing architectures to the different activities of the analytics data value chain (DVC). We find out in those infrastructures that data organization and decision-making support have been under-explored so far. Based on the lessons learnt from the review, we also identify that design tensions like architecture distribution, flexibility and extensibility (and an increased focus on data organization and decision making) are some of the most promising issues to be addressed by the MMLA community in the near future.

In blended learning scenarios, evidence needs to be gathered from digital and physical spaces to obtain a more complete view of the teaching and learning processes. However, these scenarios are highly heterogeneous, and the varying data sources available in each particular context can condition the accuracy, relevance, interpretability and actionability of the Learning Analytics (LA) solutions, affecting also the user's sense of agency and trust in such solutions. To aid stakeholders in making use of learning analytics, we propose a process to involve teachers in customizing multimodal LA (MMLA) solutions, adapting them to their particular blended learning situation (e.g., identifying relevant data sources and metrics). Since measuring the added value of adopting an LA solution is not straightforward, we also propose a concrete method for doing so. The results obtained from two case studies in authentic, blended computer-supported collaborative learning settings show an improvement in the sensitivity and F1 scores of the customized MMLA solution. Aside from these quantitative improvements, participant teachers reported both an increment in the effort involved, but also increased relevance, understanding and actionability of the results.

Despite the educational affordances that ubiquitous learning has shown, it is still hampered by several orchestration difficulties. One of these difficulties is that teachers lose awareness of what the students perform across the multiple technologies and spaces involved. Monitoring can help in such awareness, and it has been highly explored in face-to-face and blended learning. Nevertheless, in ubiquitous learning environments, monitoring has been usually limited to activities taking place in a specific type of space (e.g., outdoors). In this article, we propose a monitoring system for ubiquitous learning, which was evaluated in three authentic studies, supporting the participants in the affordable monitoring of learning situations involving web, augmented-physical, and 3D virtual world spaces. The work carried out also helped identify a set of guidelines, which are expected to be useful for researchers and technology developers aiming to provide participants’ support in ubiquitous learning environments.

This work in progress proposes a Smart Schoolhouse approach to STEM education in primary schools through engaging learners in inquiry on their surroundings in a data-rich physical and digital environment. The paper summarizes the process and outcomes of an interview with a group of experts on requirements assessment for setting up data-rich school buildings. Experts from various fields (business informatics, computer science, electronics, learning analytics and STEM pedagogy) were engaged in a focus group with a twofold purpose: to identify the setbacks of using different sensors for data collection in the physical learning environment; and to classify the possible types of data collection technologies. A small rural school has been chosen for piloting the proposed technological and pedagogical setup as an experimental school providing in-depth inquiry-based learning of STEM subjects, using the schoolhouse itself as an object and an instrument of study.

Multimodal Learning Analytics (MMLA) captures, integrates and analyzes learning traces from different sources in order to obtain a more holistic understanding of the learning process, wherever it happens. MMLA leverages the increasingly widespread availability of diverse sensors, high-frequency data collection technologies and sophisticated machine learning and artificial intelligence techniques. The aim of this workshop is twofold: first, to expose participants to, and develop, different multimodal datasets that reflect how MMLA can bring new insights and opportunities to investigate complex learning processes and environments; second, to collaboratively identify a set of grand challenges for further MMLA research, built upon the foundations of previous workshops on the topic.

'Teaching analytics' is the application of learning analytics techniques to understand teaching and learning processes, and eventually enable supportive interventions. However, in the case of (often, half-improvised) teaching in face-to-face classrooms, such interventions would require first an understanding of what the teacher actually did, as the starting point for teacher reflection and inquiry. Currently, such teacher enactment characterization requires costly manual coding by researchers. This paper presents a case study exploring the potential of machine learning techniques to automatically extract teaching actions during classroom enactment, from five data sources collected using wearable sensors (eye-tracking, EEG, accelerometer, audio and video). Our results highlight the feasibility of this approach, with high levels of accuracy in determining the social plane of interaction (90%, κ=0.8). The reliable detection of concrete teaching activity (e.g., explanation vs. questioning) accurately still remains challenging (67%, κ=0.56), a fact that will prompt further research on multimodal features and models for teaching activity extraction, as well as the collection of a larger multimodal dataset to improve the accuracy and generalizability of these methods.

Research on learning dashboards aims to identify what data is meaningful to different stakeholders in education, and how data can be presented to support sense-making processes. This paper summarizes the main outcomes of a systematic literature review on learning dashboards, in the fields of Learning Analytics and Educational Data Mining. The query was run in five main academic databases and enriched with papers coming from GScholar, resulting in 346 papers out of which 55 were included in the final analysis. Our review distinguishes different kinds of research studies as well as different aspects of learning dashboards and their maturity in terms of evaluation. As the research field is still relatively young, many of the studies are exploratory and proof-of-concept. Among the main open issues and future lines of work in the area of learning dashboards, we identify the need for longitudinal research in authentic settings, as well as studies that systematically compare different dashboard design options.