Collaborative Problem Solving (CPS) has emerged as a key competence for the 21st century. In support of this, valid assessments of CPS skills have become critical. However, limited research has designed and developed CPS assessments for young students. Based on multimodal learning analytics, we aim to develop and validate a game-based assessment of CPS for primary school students. In this study, evidence centered design approach was used to design and develop the game-based CPS assessment. Specifically, we designed and developed a mobile multiplayer online 3D role-playing game on CPS and a coding scheme for coding students’ gameplay data (i.e., game logs and voice chat) based on the ATC21S CPS framework. A total of 32 primary 5 students participated in this study to play the game in a group of four and complete a questionnaire of CPS skills. The gameplay data were coded based on our coding scheme. Correlation analysis between the coded results and the CPS questionnaire data supported the criterion validity of our game-based assessment measure. Additionally, the results of expert interview facilitated our understanding of assessment design and data use. This study will make methodological and practical contributions to the integration of MMLA into game-based CPS assessments.

Well-designed digital games can serve as the vehicle to assess and support young people’ collaborative problem solving (CPS) skills. However, there is limited research leveraging multimodal learning analytics (MmLA) to explore students’ game-based CPS processes and outcomes. Inspired by MmLA methods and approaches, this preliminary study aims to examine students’ demonstration of CPS skills through collecting and analyzing a dataset of combined game logs and verbal discourses from two groups of primary school students with contrasting performances. Based on the Assessment and Teaching of 21st Century Skills CPS framework, we iteratively coded the dataset. Results of descriptive statistics showed that the successful group exhibited cognitive skills more frequently while the unsuccessful group showcased social skills more. Results of epistemic network analysis (ENA) revealed that, in both social and cognitive dimensions, the successful group demonstrated more diverse and stronger associations among various subskills, whereas there were fewer associations in the unsuccessful group. Implications are drawn for MmLA and CPS research and teaching practices of CPS skills.

In this paper, we present an empirical study on collaborative virtual reality (VR) maker activities. A platform called CLEVR was designed to facilitate real-time VR co-creation with group awareness features. We develop an assessment framework for collaborative quality using log-based learning analytics (LA). By conducting time-series analysis on five pairs of participants, we demonstrate the potential of our platform and methodology for the development of LA tools to understand and support collaborative learning.

Widely accessible online courses provide a feasible platform for teachers' continuous online professional learning. The learning analytics dashboard (LAD) provides fine-grained and actionable feedback that supports learners’ self-regulated learning. However, previous studies on LAD design and evaluation predominantly focused on student-facing LADs, with scarce attention on LADs designed for teacher-learners. This study introduces the LAD in an online learning platform for college teachers and conducts a preliminary evaluation with 18 participants. Results show their largely positive ratings on five criteria (e.g., perceived usefulness, ease of use, and behavioral changes) and offer feedback for further refinements of the LAD. This study will improve our understanding of LA-enabled teacher online professional learning and provide practical implications for designing and evaluating LA tools catered to teacher-learners.

Previous studies have evaluated the affordances and challenges of performing cognitively demanding learning tasks with background music (BGM), yet the effects of various types of BGM on learning still remain an open question. This study aimed to examine the impacts of different music genres and fine-grained music characteristics on learners’ emotional, physiological, and pupillary responses during reading comprehension. Leveraging multimodal learning analytics (MmLA) methods of collecting data in multiple modalities from learners, a user experiment was conducted on 102 participants, with half of them reading with self-selected BGM (i.e., the experimental group), while the other half reading without BGM (i.e., the control group). Results of statistical analyses and interviews revealed significant differences between the two groups in their self-reported emotions and automatically measured physiological responses when the experimental group was exposed to classical, easy-listening, rebellious and rhythmic music. Fine-grained music characteristics (e.g., instrumentation, tempo) could predict learners’ emotions, pupillary, and physiological responses during reading comprehension. The expected contributions of this study include: 1) providing empirical evidence for understanding affective dimensions of learning with BGM, 2) applying MmLA methods for examining the impacts of BGM on learning, and 3) yielding practical implications on how to improve learning with BGM.

Online courses enable wide access to educational resources and thus provide a feasible platform for cross-regional teacher professional learning. Learning analytics dashboards (LAD) can support online learners by providing fine-grained feedback generated from learners’ interactions with platforms. Nevertheless, most studies on teacher online professional learning focus on resource-rich and technology-advanced regions, with scarce attention to the Global South. Furthermore, existing studies on LAD design mainly target students’ learning, rather than teachers’ professional learning. Therefore, it is much needed to develop LAD for teacher-learners online professional learning in the Global South. Contextualized in an international online professional training initiative, this study conducted in-depth interviews with 42 teacher-learners from 19 countries in the Global South, aiming to identify their needs for 1) support on their self-regulated learning (SRL), and 2) potential LA components in dashboards. Findings indicated that teacher-learners needed support for self-regulated learning strategies, including motivation maintenance, time management, environment structuring, help-seeking, and self-evaluation. Nine LA features were identified to design the LADs to support SRL preliminarily. This co-designed LAD study with interviewees improved our understanding on the needs of college teachers in the Global South for LA support during their online professional learning, generating practical insights into needs-driven LAD designs.

While current evaluation of maker activities has rarely explored students’ learning processes, the multi-perspective and multi-level nature of collaboration adds complexity to learning processes of collaborative maker activities. In terms of group dynamics as an important indicator of collaboration quality, extant studies have shown the benefits of synchrony between learners’ actions during collaborative learning processes. However, synchrony of learners’ cognitive processes and visual attention in collaborative maker activities remains under-explored. Leveraging the multimodal learning analytics (MMLA) approach, this pilot study examines learners’ synchrony patterns from multiple modalities of data in the collaborative maker activity of virtual reality (VR) content creation. We conducted a user experiment with five pairs of students, and collected and analyzed their electroencephalography (EEG) signals, eye movement and system log data. Results showed that the five pairs of collaborators demonstrated diverse synchrony patterns. We also discovered that, while some groups exhibited synchrony in one modality of data before becoming not synchronized in another modality, other groups started with a lack of synchrony followed by maintaining synchrony. This study is expected to make methodological and practical contributions to MMLA research and assessment of collaborative maker activities.

Assessments of digital maker activities increasingly rely on automatically analyzing student-created products and their components, such as their textual output. In particular, recent learning analytics research has proposed incorporating text analytic feedback for facilitating students' virtual reality (VR) content creation, though lacking direct empirical evidence from student-created artefacts. Thus, this study examined the relationships between metrics on text in student-created VR content and their learning performance. VR narration scripts and performance scores were collected from 102 students in a maker-based general education course. Results of statistical testing and text mining show that high and low-performing students demonstrated significant differences in such metrics as word counts, vocabulary sizes, and frequent unigrams and bigrams. This study makes methodological and practical contributions in the domains of maker education and learning analytics.

Existing learning analytics (LA) studies on self-regulated learning (SRL) have rarely focused on maker education that emphasizes student autonomy in their learning process. Towards using LA methods for generating evidence of SRL in maker-based courses, this study leverages logs of a learning management system (LMS) with its activity design aligned with the maker-based pedagogy. We explored frequencies and sequential patterns of students’ SRL behaviors as reflected in the LMS logs and their relations with learning performance. Adopting a mixed method approach, we collected and triangulated both quantitative (i.e., system logs, performance scores) and qualitative (i.e., student-written reflections) data sources from 104 students. Based on current LA-based SRL research, we developed an LMS log-based analytic framework to define the SRL phases and behaviors applicable to maker activities. Statistical, data mining, and qualitative analysis methods were conducted on 48,602 logged events and 131 excerpts extracted from student reflections. Results reveal that high-performing students demonstrated some SRL behaviors (e.g., Making Personal Plans, Evaluation) more frequently than their low-performing counterparts, yet the two groups showcased fairly similar sequences of SRL behaviors. Theoretical, methodological and pedagogical implications are drawn for LA-based SRL research and maker education.

Due to the popularity of virtual reality (VR) in education settings and the rise of maker education, this paper presents LAVR, a platform for VR content creation with learning analytics functions. We design the platform where students can easily create VR stories through a web interface. A learning analytics dashboard is implemented to provide students with feedback on their progress and the quality of the textual content in their VR stories. The platform also offers learning management features for helping teachers set up classrooms with assignments. While the platform will be employed in a forthcoming general education course, we have conducted a preliminary usability evaluation with 12 students and one teacher, and gathered feedback for further refinements before its official launch. The platform will contribute to integrating learning analytics with maker activities.

In times of pandemic-induced challenges, virtual reality (VR) allows audience to learn about cultural heritage sites without temporal and spatial constraints. The design of VR content is largely determined by professionals, while evaluations of content often rely on learners’ self-report data. Learners’ attentional focus and understanding of VR content might be affected by the presence or absence of different multimedia elements including text and audio-visuals. It remains an open question which design variations are more conducive for learning about heritage sites. Leveraging eye-tracking, a technology often adopted in recent multimodal learning analytics (MmLA) research, we conducted an experiment to collect and analyze 40 learners’ eye movement and self-reported data. Results of statistical tests and heatmap elicitation interviews indicate that 1) text in the VR environment helped learners better understand the presented heritage sites, regardless of having audio narration or not, 2) text diverted learners’ attention away from other visual elements that contextualized the heritage sites, 3) exclusively having audio narration best simulated the experience of a real-world heritage tour, 4) narration accompanying text prompted learners to read the text faster. We make recommendations for improving the design of VR learning materials and discuss the implications for MmLA research.

Being a promising constructionist pedagogy in recent years, maker education empowers students to take agency of their learning process through constructing both knowledge and real-world physical or digital products and fosters peer interactions for collective innovation. Learning Analytics (LA) excels at generating personalized, fine-grained feedback in near real-time and holds much potential in supporting process-oriented and peer-supported learning activities, including maker activities. In the context of virtual reality (VR) content creation for cultural heritage education, this study qualitatively solicited 27 students’ needs on progress monitoring, reflection, and feedback during their making process. Findings have inspired the prototype design of a student-facing LA dashboard (LAVR). Leveraging multimodal learning analytics (MmLA) such as text and audio analytics to fulfill students’ needs, the prototype has various features and functions including automatic task reminders, content quality detection, and real-time feedback on quality of audio-visual elements. A preliminary evaluation of the prototype with 10 students confirms its potential in supporting students’ self-regulated learning during the making process and for improving the quality of VR content. Implications on LA design for supporting maker education are discussed. Future work is planned to include implementation and evaluation of the dashboard in classrooms.

Learning analytics (LA) mines, analyzes and visualizes the data of students' learning behaviours on learning platforms such as Learning Management Systems (LMS), but few LA tools built are adaptable to multiple platforms or for general education courses. This study sets out to evaluate a lightweight LA tool implemented on Moodle and Open edX for monitoring students' learning progress. Survey data were collected from 156 students, supplemented by interview responses from 25 students and three instructors. Preliminary results show that a considerable portion of surveyed students used the LA tool and they held positive opinions on its efficacy in monitoring self-progress and the effectiveness of its visualizations for information delivery. Nonetheless, learners who did not use the LA tool raised concerns about it relying only on their online behaviours without considering their offline learning. Coupled with instructors' evaluation results, discussion and implications are presented.

Through profiling learners’ music usage in everyday learning settings and depicting their learning experience when studying with a music app powered by a large-scale and real-world music library, this study revealed preliminary observations on how background music impacts learning under varying task load, and manifested intriguing patterns of learners’ music usage and music preferences in various task load conditions. Specifically, we piloted a three-day field experiment in students’ everyday learning environment. During the experiment, participants performed learning tasks with music in the background and completed a set of online surveys before and after each learning session. Our results suggested that learners’ self-selected, real-life background music could enhance their learning effectiveness, while the beneficial effect of background music was more apparent when the learning task was less mentally or temporally demanding. Towards a closer look at the characteristics of preferable music pieces under various task load conditions, our findings showed that music preferred by participants under high versus low temporal demand differs in a number of characteristics, including speechiness, acousticness, danceability, and energy. This study further reveals the effects of background music on learning under varying task load levels and provides implications for context-aware background music selection when designing musically enriched learning environments.

Digitization provides a solution for documentation and preservation of nonmovable cultural heritages. Despite efforts for the preservation of cultural heritages around the world, no well‐accepted metadata schema has been developed for murals and stone cave temples, which are often high‐value heritages built in ancient times. In addition, the literature is scarce on the user‐centered evaluation of metadata schemas of this kind. This study therefore aims to offer insights on developing and evaluating a metadata schema for organizing information of these historic and complex cultural heritages. In‐depth interviews were conducted with a total of 30 users, including 18 professional and 12 public users, and interview transcripts were coded through a qualitative content analysis approach. Findings reveal the importance of specific metadata elements as perceived by the two groups of end users, which correlated with their cultural heritage information‐seeking behaviors. In addition, the issues of standardization of cataloging of cultural heritage information and interoperability among metadata schemas have been raised by users for enhancing the user experience with digital platforms of cultural heritage information. The coding schema developed in this study can serve as a framework for follow‐up evaluations of metadata schemas, contributing to the ongoing development of cultural heritage metadata.

This poster presents a cross-platform learning analytics dashboard on Moodle and Open edX for monitoring outcome-based learning progress. The dashboard visualizes students' interactions with the platforms in near real-time, aiming to help teachers and students monitor students' learning progress. The dashboard has been used in four large-size general education courses in a comprehensive university in Hong Kong, undergoing evaluation and improvement.

This study attempts to investigate to what extent indicators of academic writing and cognitive thinking can help measure the writing quality of group collaborative writings on Wikis. Particularly, comparisons were made on Wiki content in different stages of the projects. Preliminary results from a multiple linear regression analysis reveal that linguistic indicators such as engagement markers and self-mention were significant predictors in earlier stages to the projects, whereas verbs indicating cognitive thinking in the evaluation level were significant in later project stages.