Embodied learning offers new opportunities to enhance learning effectively, and engage children with stimulating educational experiences. Multi Sensory Environments (MSEs) are spaces that allow for several interaction modalities that stimulate users’ senses and allow the collection of multimodal data. In educational contexts, they provide opportunities to support children’s learning in a playful manner. The use of MSEs is usually carried out with the collaboration of teachers; their perspectives and responsibilities are crucial for the children’s experience. The goal of our research is to uncover evidence-based challenges and opportunities, while considering teachers’ experiences. We conducted fourteen semi-structured interviews with teachers (n = 6) and researchers (n = 8) experienced using MSEs’, and analysed the identified challenges and considerations during a workshop with four Child-Computer Interaction (CCI) experts. We offer a series of implications for consideration when designing and/or using MSEs to support children’s learning.

Identifying and supporting children’s play and problem solving behaviour is important for designing educational technologies. This can inform feedback mechanisms to scaffold learning (provide hints or progress information), and assist facilitators (teachers, parents) in supporting children. Traditionally, researchers manually code video to dissect children’s nuanced play and problem solving behaviour. Advancements in sensing technologies and their respective Multi-Modal Data (MMD), afford observation of invisible states (cognitive, affective, physiological), and provide opportunities to inspect internal processes experienced during learning and play. However, limited research combines traditional video annotations and MMD to understand children’s behaviour as they interact with educational technology. To address this concern, we collected data from webcam, wristband, eye-trackers, and Kinect, as 26 children, aged 10-12, played a Motion-Based Educational Games (MBEG). Results showed significant differences in children’s experience during play and problem solving episodes, and motivate design considerations aimed to facilitate children’s interactions with MBEG.

This paper investigates the relation between children’s joint gaze and emotions with the information flow of the screen from a causal point of view, in the context of collaborative coding. We employ Granger’s definition of causality to extend the knowledge we have about children’s collaborative activities from correlational methods. We organised a coding workshop with 50 children (10 dyads and 10 triads; 13-16 years old). While the children were coding collaboratively, their facial video and the screen were recorded. From the screen recording we computed the information flow; and from the facial video we computed children’s emotions (e.g., frustration and boredom) and estimated their gaze. The gaze estimation was used to compute the joint visual attention (JVA) of the team. Our results show that for high performing teams JVA drives the information flow; while for low performing teams we observe causal relation between emotions and information flow. In particular for the low performing teams, frustration and boredom drive the information flow and the information flow then drives children’s confusion. These results extend the understanding of the socio-cognitive processes underlying collaborative performance, which is primarily correlational in nature, with the causal relations between measurements. These novel results have the potential to guide the design of learning tools that scaffold children’s learning and collaboration.

Abstract Embodied interaction describes the interplay between the brain and the body and its influence on the sharing, creation and manipulation of meaningful interactions with technology. Spatial skills entail the acquisition, organization, utilization and revision of knowledge about spatial environments. Embodied interaction is a rapidly growing topic in human–computer interaction with the potential to amplify human interaction and communication capacities, while spatial skills are regarded as key enablers for the successful management of cognitive tasks. This work provides a systematic review of empirical studies focused on embodied interaction and spatial skills. Thirty-six peer-reviewed articles were systematically collected and analysed according to their main elements. The results summarize and distil the developments concerning embodied interaction and spatial skills over the past decade. We identify embodied interaction capacities found in the literature review that help us to enhance and develop spatial skills. Lastly, we discuss implications for research and practice and highlight directions for future work.

Motion-Based Touchless Games (MBTG) are being investigated as a promising interaction paradigm in children's learning experiences. Within these games, children's digital persona (i.e, avatar), enables them to efficiently communicate their motion-based interactivity. However, the role of children's Avatar Self-Representation (ASR) in educational MBTG is rather under-explored. We present an in-situ within subjects study where 46 children, aged 8--12, played three MBTG with different ASRs. Each avatar had varying visual similarity and movement congruity (synchronisation of movement in digital and physical spaces) to the child. We automatically and continuously monitored children's experiences using sensing technology (eye-trackers, facial video, wristband data, and Kinect skeleton data). This allowed us to understand how children experience the different ASRs, by providing insights into their affective and behavioural processes. The results showed that ASRs have an effect on children's stress, arousal, fatigue, movement, visual inspection (focus) and cognitive load. By exploring the relationship between children's degree of self-representation and their affective and behavioural states, our findings help shape the design of future educational MBTG for children, and emphasises the need for additional studies to investigate how ASRs impacts children's behavioural, interaction, cognitive and learning processes.

Multi-Modal Data (MMD) can help educational games researchers understand the synergistic relationship between player's movement and their learning experiences, and consequently uncover insights that may lead to improved design of movement-based game technologies for learning. Predicting player performance fosters opportunities to cultivate heightened educational experiences and outcomes. However, predicting player's performance utilising player-generated MMD during their interactions with educational Motion-Based Touchless Games (MBTG) is challenging. To bridge this gap, we implemented an in-situ study where 26 users, age 11, played 2 maths MBTGs in a single 20-30 minute session. We collected player's MMD (i.e., gaze data from eye-tracking glasses, physiological data from wristbands, and skeleton data from Kinect) produced during game-play. To investigate the potential of MMD for predicting player's academic performance, we used machine learning techniques and MMD derived from player's game-play. This allowed us to identify the MMD features that drive rapid highly accurate predictions of players' academic performance in educational MBTGs. This might allow us to provide real-time proactive feedback to the player to support them through their educational gaming experience. Our analysis compared two data lengths corresponding to half and full duration of the player's question solving time. We showed that all combinations of extracted features associated with gaze, physiological, and skeleton data, predicted student performance more accurately than the majority baseline. Additionally, the most accurate prediction of player's performance derived from the combination of gaze and physiological data for both full and half data lengths. Our findings emphasise the significance of using MMD for real-time performance prediction in educational MBTG and offer implications for practice.

Embodied Interaction (EI) offers unique opportunities to uncover novel ways to achieve experiential learning whilst keeping students stimulated and engaged. Spatial abilities have been repeatedly demonstrated as a success predictor for educations and professions in Science, Technology, Engineering and Mathematics. However, many researchers argue that training and assessment of this pertinent reasoning skill is vastly underrepresented in the school curriculum. This paper presents TetRotation, a PhD centred on how affordances coming from Multimodal Analytics can be coupled with EI to nurture Mental Rotation (MR) skills. The overarching objectives of the project are two fold. First, the TetRotation Interaction Design study will highlight best practices identified through the assessment of efficiency, level of engagement and learning gains achieved when using gesture based EI to solve MR tasks. Next, in the TetRotation Game study, these design practices will guide the implementation of an interactive serious game purposed to support the development of MR skills. This research relies on mixed method techniques, including data collections from users' actions, like motion sensing, EEG, gaze tracking, video-recordings, click streams, interviews and surveys.