Multimodal learning analytics (MMLA) research for building collaboration quality estimation models has shown significant progress. However, the generalizability of such models is seldom addressed. In this paper, we address this gap by systematically evaluating the across-context generalizability of collaboration quality models developed using a typical MMLA pipeline. This paper further presents a methodology to explore modelling pipelines with different configurations to improve the generalizability of the model. We collected 11 multimodal datasets (audio and log data) from face-to-face collaborative learning activities in six different classrooms with five different subject teachers. Our results showed that the models developed using the often-employed MMLA pipeline degraded in terms of Kappa from Fair (.20 < Kappa < .40) to Poor (Kappa < .20) when evaluated across contexts. This degradation in performance was significantly ameliorated with pipelines that emerged as high-performing from our exploration of 32 pipelines. Furthermore, our exploration of pipelines provided statistical evidence that often-overlooked contextual data features improve the generalizability of a collaboration quality model. With these findings, we make recommendations for the modelling pipeline which can potentially help other researchers in achieving better generalizability in their collaboration quality estimation models.

Multimodal Learning Analytics (MMLA) has been applied to collaborative learning, often to estimate collaboration quality with the use of multimodal data, which often have uneven time scales. The difference in time scales is usually handled by dividing and aggregating data using a fixed-size time window. So far, the current MMLA research lacks a systematic exploration of whether and how much window size affects the generalizability of collaboration quality estimation models. In this paper, we investigate the impact of different window sizes (e.g., 30 seconds, 60s, 90s, 120s, 180s, 240s) on the generalizability of classification models for collaboration quality and its underlying dimensions (e.g., argumentation). Our results from an MMLA study involving the use of audio and log data showed that a 60 seconds window size enabled the development of more generalizable models for collaboration quality (AUC 61%) and argumentation (AUC 64%). In contrast, for modeling dimensions focusing on coordination, interpersonal relationship, and joint information processing, a window size of 180 seconds led to better performance in terms of across-context generalizability (on average from 56% AUC to 63% AUC). These findings have implications for the eventual application of MMLA in authentic practice.

Multimodal Learning Analytics (MMLA) aims to support evidence-based educational practices by collecting, processing, analyzing and sense-making of multimodal evidence of learning. MMLA is not widespread yet and there are few tailored MMLA solutions to meet the requirements of a specific learning scenario. This PhD project investigates the main challenges behind the limited development of MMLA solutions and proposes an MMLA infrastructure as the main contribution. The proposed infrastructure includes three components - a data value chain, a data model and a software architecture. The overall project follows the design-based research methodology where multiple iterations are involved to refine the contributions.

Collaborative learning flow patterns (CLFPs) encode solutions to recurrent pedagogical problems, which have been successfully applied to the design of learning experiences. However, the pedagogical knowledge encoded in these patterns has seldom been exploited in learning analytics (LA). This paper analyzes four of the most common CLFPs to extract the intrinsic constraints that lead to a successful collaborative learning activity, and use them to enhance existing LA solutions. To understand the added value of applying such codified knowledge in LA, we present evidence from five authentic case studies in which such constraints aided university teachers in monitoring complex collaborative scripts. The results not only illustrate quantitatively such added value but also unearth qualitative benefits, such as raising practitioners' awareness about how the current state of activities may affect future phases of the script.

The collection of evidence from authentic classroom practice is important both for teacher professional development (TPD) and educational research. However, most current learning analytics (LA) fails to capture the physical occurrences of the classroom, and do not always address individual teachers' needs. More customizable forms of classroom data collection (e.g., through video-recordings or by human observers) are time-consuming and expensive to implement. Navigating this tradeoff between the personalization of data collection and the strict time/effort constraints of classroom practice is still an unsolved challenge for designers of LA and teaching analytics (TA) systems to be used in face-to-face classrooms. In this paper, we extract lessons learnt from a design-based research process that explores this tradeoff, towards the development of teacher-driven, personalizable data collection tools. Through a survey of 15 expert teachers and paper and software prototype testing with a total of 14 teachers in simulated and authentic settings, we gathered information about teachers' preferences for teacher-driven data collection in classrooms, and derived design insights (e.g., a cold-start problem and the need for multiple layers in the personalization), which can be useful for the design of TA/LA tools that collect personalized data from everyday classrooms.

There is an increasing interest in Multimodal Learning Analytics (MMLA), which involves complex technical issues in gathering, merging and analyzing different types of learning data from heterogeneous data sources. However, there is still no common reference architecture to face these technical challenges of MMLA. This paper summarizes the state of the art of MMLA software architectures through a systematic literature review. Our analysis of nine architecture proposals highlights the uneven support provided by existing architectures to the different activities of the analytics data value chain (DVC). We find out in those infrastructures that data organization and decision-making support have been under-explored so far. Based on the lessons learnt from the review, we also identify that design tensions like architecture distribution, flexibility and extensibility (and an increased focus on data organization and decision making) are some of the most promising issues to be addressed by the MMLA community in the near future.

In blended learning scenarios, evidence needs to be gathered from digital and physical spaces to obtain a more complete view of the teaching and learning processes. However, these scenarios are highly heterogeneous, and the varying data sources available in each particular context can condition the accuracy, relevance, interpretability and actionability of the Learning Analytics (LA) solutions, affecting also the user's sense of agency and trust in such solutions. To aid stakeholders in making use of learning analytics, we propose a process to involve teachers in customizing multimodal LA (MMLA) solutions, adapting them to their particular blended learning situation (e.g., identifying relevant data sources and metrics). Since measuring the added value of adopting an LA solution is not straightforward, we also propose a concrete method for doing so. The results obtained from two case studies in authentic, blended computer-supported collaborative learning settings show an improvement in the sensitivity and F1 scores of the customized MMLA solution. Aside from these quantitative improvements, participant teachers reported both an increment in the effort involved, but also increased relevance, understanding and actionability of the results.

In this paper we discuss work in progress regarding the challenges in education (generation z, smart devices, etc.), especially at technical universities where the teaching staff lack or have very little pedagogical background. One potential way of aiding the instructors who want to improve their teaching practice is through the recording and observation of their own classroom activities (e.g., using video or other sensors), which would enable lecturers analyse their teaching and make informed decisions about any necessary changes in their teaching methods. The present article will give a short overview of related work (teaching analytics and instruments currently used in recording classroom management) and share the steps taken in implementing the approach at Tallinn University of Technology (TUT), Estonia.

This paper describes a Learning Design (LD) tool, aimed at supporting teachers' conceptualization of collaborative learning activities for students. The main element of innovation of the tool, called the "augmented 4Ts game", in respect to the other existing LD tools, lays in its being half-tangible-half-digital. The tangible component of the game is based on a paper board and a set of cards, and it is meant to lower teachers' typical barriers towards technology-based tools, and to make the game more flexible and engaging. At the same time, the digital component allows to continue the design work remotely beyond face-to-face sessions, and opens the way for further planning, delivering and sharing support.

Multimodal Learning Analytics (MMLA) captures, integrates and analyzes learning traces from different sources in order to obtain a more holistic understanding of the learning process, wherever it happens. MMLA leverages the increasingly widespread availability of diverse sensors, high-frequency data collection technologies and sophisticated machine learning and artificial intelligence techniques. The aim of this workshop is twofold: first, to expose participants to, and develop, different multimodal datasets that reflect how MMLA can bring new insights and opportunities to investigate complex learning processes and environments; second, to collaboratively identify a set of grand challenges for further MMLA research, built upon the foundations of previous workshops on the topic.

'Teaching analytics' is the application of learning analytics techniques to understand teaching and learning processes, and eventually enable supportive interventions. However, in the case of (often, half-improvised) teaching in face-to-face classrooms, such interventions would require first an understanding of what the teacher actually did, as the starting point for teacher reflection and inquiry. Currently, such teacher enactment characterization requires costly manual coding by researchers. This paper presents a case study exploring the potential of machine learning techniques to automatically extract teaching actions during classroom enactment, from five data sources collected using wearable sensors (eye-tracking, EEG, accelerometer, audio and video). Our results highlight the feasibility of this approach, with high levels of accuracy in determining the social plane of interaction (90%, κ=0.8). The reliable detection of concrete teaching activity (e.g., explanation vs. questioning) accurately still remains challenging (67%, κ=0.56), a fact that will prompt further research on multimodal features and models for teaching activity extraction, as well as the collection of a larger multimodal dataset to improve the accuracy and generalizability of these methods.

Research on learning dashboards aims to identify what data is meaningful to different stakeholders in education, and how data can be presented to support sense-making processes. This paper summarizes the main outcomes of a systematic literature review on learning dashboards, in the fields of Learning Analytics and Educational Data Mining. The query was run in five main academic databases and enriched with papers coming from GScholar, resulting in 346 papers out of which 55 were included in the final analysis. Our review distinguishes different kinds of research studies as well as different aspects of learning dashboards and their maturity in terms of evaluation. As the research field is still relatively young, many of the studies are exploratory and proof-of-concept. Among the main open issues and future lines of work in the area of learning dashboards, we identify the need for longitudinal research in authentic settings, as well as studies that systematically compare different dashboard design options.

Recent technological advances in mobile devices enable the connection of classrooms with other virtual and physical spaces. Some approaches aim at helping teachers carry out learning situations across such spaces. However, these proposals tend to be isolated from other activities in teachers’ current common practices, and do not allow teacher-controlled flexibility of what students do during the enactment. Aiming to overcome such limitations, the Bucket-Server is a system that enables teachers to include learning buckets in their learning situations: containers of learning artifacts generated and/or consumed across-spaces by students during the enactment. Teachers create learning buckets at design time, configuring them with constraints to regulate the degree of freedom offered to the students. These learning buckets can be integrated into multiple existing technologies used in different educational spaces (e.g., web, physical and 3D virtual world spaces), thus helping embed buckets in the teachers’ current common practices.

Adaptive Collaborative Scripting systems provide learning benefits by adapting leaner scaffolding to the students and their current context. However, their development is still in its infancy and they are not widespread in the Technology Enhanced Learning TEL practice, which often uses VLEs like Moodle and other Web 2.0 tools. In order to assess the feasibility of applying the ACS approach on a larger scale, this paper presents the initial results of a short-term evaluation of the GLUE! suite of tools. The main goal of this specific evaluation process was to identify possible opportunities and ideas on how to design and deploy adaptive Computer- Supported Collaborative Learning (CSCL) activities using widespread VLEs and Web 2.0 tools in order to maximise community acceptance and lower development efforts. The main findings of the evaluation provide incentive to further explore both the impact and the complexity of the design and the deployment of adaptive collaboration scripts.