Multimodal learning analytics (MMLA) research for building collaboration quality estimation models has shown significant progress. However, the generalizability of such models is seldom addressed. In this paper, we address this gap by systematically evaluating the across-context generalizability of collaboration quality models developed using a typical MMLA pipeline. This paper further presents a methodology to explore modelling pipelines with different configurations to improve the generalizability of the model. We collected 11 multimodal datasets (audio and log data) from face-to-face collaborative learning activities in six different classrooms with five different subject teachers. Our results showed that the models developed using the often-employed MMLA pipeline degraded in terms of Kappa from Fair (.20 < Kappa < .40) to Poor (Kappa < .20) when evaluated across contexts. This degradation in performance was significantly ameliorated with pipelines that emerged as high-performing from our exploration of 32 pipelines. Furthermore, our exploration of pipelines provided statistical evidence that often-overlooked contextual data features improve the generalizability of a collaboration quality model. With these findings, we make recommendations for the modelling pipeline which can potentially help other researchers in achieving better generalizability in their collaboration quality estimation models.

During collaborative learning (CL) teachers have little to no information about what is happening in the groups, however, they are expected to plan, monitor, support, consolidate and reflect upon student interactions. How is a teacher able to provide support the students when she/he is not fully aware of the situation and progress of the students? Multimodal learning analytics (MMLA) could offer teachers valuable insights into the CL process, however, not many MMLA outputs are used in real practice today, and designing such MMLA dashboards remains a challenge. The aim of the present design-science research is to find out if multimodal learning analytics is able to help teachers help support students in CL. After having gone through two cycles of design based research, it is apparent that the involved teachers are interested in getting an overview of the activities within the groups and help with assessment. The involved teachers were ready to use collaboration analytics if it can assure a certain level of accuracy, more research is needed on how to use MMLA to support teachers in CL assessment.