Virtual Reality (VR) enriches learning and instruction, while background music (BGM) is widely employed to modulate attention and cognition. Understanding how BGM influences learning in VR is essential for optimizing VR learning environments and has the potential to improve educational outcomes, yet this area remains largely unexplored. This study collected fifty-two participants’ self-reports, electroencephalogram signals, eye movements, heart rates, and interview responses to explore their attention and cognition during studying virtual heritage sites with and without BGM, while considering individual traits as influential factors. Results showed that with BGM, participants reported higher levels of engagement and demonstrated longer fixation duration on heritage sites’ image regions, than without BGM. Moreover, participants’ self-reported familiarity with the cultural heritage site and BGM listening frequency moderated the effect of BGM on attention and cognition in VR. This study offers implications for incorporating BGM for learning in VR, and designing personalized VR learning environments.

Many students enjoy listening to background music (BGM) when they read, but it is challenging to measure their meta-cognitive states (e.g., understanding of the passage, engagement in reading). Eye movements, as an approach in multimodal learning analytics (MmLA), can offer continuous fine-grained data that reflect learners’ cognitive processes. This study explored the potential of utilizing eye movement measures to predict learners’ meta-cognition during reading with BGM. Results showed that learners’ eye movement measures integrated with the characteristics of the BGM, learner traits, and text complexity could predict their meta-cognitive states in reading. Findings can advance our understanding of human meta-cognition in multi-channel learning settings and provide insights for personalized BGM recommendations to enhance reading experiences.

This study employed multimodal learning analytics methods to evaluate different virtual reality (VR) interfaces, exploring their impact on the learning experience of cultural heritage. We recorded and analyzed the participants’ learning behavior, self-reported perceptions from questionnaires’ responses, electroencephalogram (EEG) signals, and visual fatigue. Preliminary results suggested that VR interfaces with more modules per scene led to higher efficiency of use, although more modules did not improve viewing experience. This study provides insights for immersive interface design in cultural heritage learning.

Previous studies have evaluated the affordances and challenges of performing cognitively demanding learning tasks with background music (BGM), yet the effects of various types of BGM on learning still remain an open question. This study aimed to examine the impacts of different music genres and fine-grained music characteristics on learners’ emotional, physiological, and pupillary responses during reading comprehension. Leveraging multimodal learning analytics (MmLA) methods of collecting data in multiple modalities from learners, a user experiment was conducted on 102 participants, with half of them reading with self-selected BGM (i.e., the experimental group), while the other half reading without BGM (i.e., the control group). Results of statistical analyses and interviews revealed significant differences between the two groups in their self-reported emotions and automatically measured physiological responses when the experimental group was exposed to classical, easy-listening, rebellious and rhythmic music. Fine-grained music characteristics (e.g., instrumentation, tempo) could predict learners’ emotions, pupillary, and physiological responses during reading comprehension. The expected contributions of this study include: 1) providing empirical evidence for understanding affective dimensions of learning with BGM, 2) applying MmLA methods for examining the impacts of BGM on learning, and 3) yielding practical implications on how to improve learning with BGM.

While current evaluation of maker activities has rarely explored students’ learning processes, the multi-perspective and multi-level nature of collaboration adds complexity to learning processes of collaborative maker activities. In terms of group dynamics as an important indicator of collaboration quality, extant studies have shown the benefits of synchrony between learners’ actions during collaborative learning processes. However, synchrony of learners’ cognitive processes and visual attention in collaborative maker activities remains under-explored. Leveraging the multimodal learning analytics (MMLA) approach, this pilot study examines learners’ synchrony patterns from multiple modalities of data in the collaborative maker activity of virtual reality (VR) content creation. We conducted a user experiment with five pairs of students, and collected and analyzed their electroencephalography (EEG) signals, eye movement and system log data. Results showed that the five pairs of collaborators demonstrated diverse synchrony patterns. We also discovered that, while some groups exhibited synchrony in one modality of data before becoming not synchronized in another modality, other groups started with a lack of synchrony followed by maintaining synchrony. This study is expected to make methodological and practical contributions to MMLA research and assessment of collaborative maker activities.

Reading is an essential medium for learning, but it is challenging to measure learners’ cognitive processes during reading. Eye-tracking, as an approach in multimodal learning analytics (MmLA), can provide fine-grained data that reflect cognitive processes during reading. In this study, we investigated whether eye movements could predict passage reading performance in addition to language proficiency and cognitive abilities. In particular, we assessed learners’ eye movement pattern and consistency through a novel method, Eye Movement analysis with Hidden Markov Models (EMHMM), in addition to traditional eye movement measures. We found that longer saccade length predicted faster reading speed Also, higher English proficiency predicted faster reading speed through the mediation of longer saccade length. In contrast, reading comprehension accuracy was best predicted by a more consistent eye fixation at the beginning of reading engagement, which may result from a better developed visual routine due to higher reading expertise. These findings have important implications for ways to assess and facilitate learners’ reading through eye movement measures and to examine factors influencing reading performance. The methods adopted could further the development of MmLA and serve as an empirical example of understanding learners’ cognitive processes through collecting and modeling critical learner-centered metrics in novel modalities.

In times of pandemic-induced challenges, virtual reality (VR) allows audience to learn about cultural heritage sites without temporal and spatial constraints. The design of VR content is largely determined by professionals, while evaluations of content often rely on learners’ self-report data. Learners’ attentional focus and understanding of VR content might be affected by the presence or absence of different multimedia elements including text and audio-visuals. It remains an open question which design variations are more conducive for learning about heritage sites. Leveraging eye-tracking, a technology often adopted in recent multimodal learning analytics (MmLA) research, we conducted an experiment to collect and analyze 40 learners’ eye movement and self-reported data. Results of statistical tests and heatmap elicitation interviews indicate that 1) text in the VR environment helped learners better understand the presented heritage sites, regardless of having audio narration or not, 2) text diverted learners’ attention away from other visual elements that contextualized the heritage sites, 3) exclusively having audio narration best simulated the experience of a real-world heritage tour, 4) narration accompanying text prompted learners to read the text faster. We make recommendations for improving the design of VR learning materials and discuss the implications for MmLA research.

Music is a popular form of entertainment and has become common practice to adjust cognition, affect, and motivation. Regarding the effects of background music on learning tasks, results are inconclusive in the literature. Recent advancement of wearable devices and computing analytics supports automated detection of multimodal physiological signals, such as eye movements, neural responses, and heart rates in a real-time fashion, which can facilitate tracking learners' changes of affect, attention, and cognition while they study with the accompaniment of background music. However, most existing studies focus only on behavioral levels, and few employed signals at physiological levels to investigate the impact of background music on learning. To fill in the research gap, this doctoral project designs two types of visual cognitive tasks, that is, reading comprehension task and art appreciation task. It aims to integrate multimodal data (e.g., eye movements, electroencephalogram (EEG), and peripheral physiological signals) to probe the effect of background music on the tasks. Its findings will extend our knowledge on the interactions among learners' performance, emotion, and engagement at both physiological and behavioral levels in multi-channel learning settings, and contribute to a goal of recommending suitable background music for self-learning.

Empirical evidence of how background music benefits or hinders learning becomes the crux of optimizing music recommendation in educational settings. This study aims to further probe the underlying mechanism through an experiment in naturalistic setting. 30 participants were recruited to join a field experiment which was conducted in their own study places for one week. During the experiment, participants were asked to conduct learning sessions with music in the background and collect music tracks they deemed suitable for learning using a novel mobile-based music discovery application. A set of participant-related, context-related, and music-related data were collected via a pre-experiment questionnaire, surveys popped up in the music app, and the logging system of the music app. Preliminary results reveal correlations between certain music characteristics and learners' task engagement and perceived task performance. This study is expected to provide evidence for understanding cognitive and emotional dimensions of background music during learning, as well as implications for the role of personalization in the selection of background music for facilitating learning.