Ongoing calls for critical data literacy emphasize the importance of designing learning opportunities for youth to build relationships with data as its author and analyst.By generating and interpreting personal and embodied data, learners can develop relational lenses about data to support these types of skills.We apply actor network theory to analyze teen interactions in a summer program on sports technologies, where youth generate, visualize, and interpret their group's personal movement data.We examine youth's arguments about what data visualizations can depict through qualitative coding of their presentations.We notice youth relying on data as an actor within their arguments and mixing the data'a actions with their remembered experiences.As collaborative actors, youth and data engage in social sensemaking with the data.We offer this frame to highlight the plurality of participation across actors.This contributes to our understanding of community data science practices for teaching and learning.We offer this as an analytical lens for researchers and educators to pay attention to, in their designs and implementations of data literacy tools and activities.Specifically, how to notice and help learners explicate the multiple actors with diverse relationships engaged in making sense of data, and how engaging youth in group, embodied data activities enables unique personal and social ways of data interpretations.

Culturally relevant and sustaining implementations of computing education are increasingly leveraging young learners' passion for sports as a platform for building interest in different STEM (Science, Technology, Engineering, and Math) concepts. Numerous disciplines spanning physics, engineering, data science, and especially AI based computing are not only authentically used in professional sports in today's world, but can also be productively introduced to introduce young learnres to these disciplines and facilitate deep engagement with the same in the context of sports. In this work, we present a curriculum that includes a constellation of proprietary apps and tools we show student athletes learning sports like basketball and soccer that use AI methods like pose detection and IMU-based gesture detection to track activity and provide feedback. We also share Scratch extensions which enable rich access to sports related pose, object, and gesture detection algorithms that youth can then tinker around with and develop their own sports drill applications. We present early findings from pilot implementations of portions of these tools and curricula, which also fostered discussion relating to the failings, risks, and social harms associated with many of these different AI methods – noticeable in professional sports contexts, and relevant to youths' lives as active users of AI technologies as well as potential future creators of the same.

In this demo, we present PaintBall, a tool that facilitates creative art creation and manipulation from movement data. It is designed as a public computation project aimed to foster conversation and connection between youth with different interests (specifically art, sports, or computing) at a community center around the shared activity of creating these rich visualizations and art pieces. We expect the design features of PaintBall – public art and tinkering work, cross interest engagement, and discrete artifact generation – to be key ideas that can be used across a variety of contexts and enable rich community development among youth by providing novel touchstones for conversation as well as reflection on these preexisting activities themselves. The following URL will host a working demo of this project – https://tiilt.northwestern.edu/projects/sportsense/paintball

This paper examines facilitator and refugee youth co-design for a summer program and asks: “how do researchers and interns understand ‘facilitation’ and ‘co-design’ in a summer program with refugee youth?” We highlight interactions with Rbekka, a refugee youth who participated in programming since 2015. We share findings that help us understand how expansive social emotional learning (SEL) and co-design create an assemblage of relationships and mutual space of learning that resists adult/child binaries and puts relational care and justice into practice.

This half-day workshop brings together practitioners, researchers, educators, and students that design, research, and engage in making activities to have fun and create artifacts using a laser cutter. Participants will engage in an interactive discussion around their own making practices and experiences. Together we aim to identify existing considerations and practices that remain to be articulated around feelings and making. That is, what does it feel or look like when making serves as a form of personal expression, or as an extension of being? How might making create and support somatic openings that frame making less as an activity toward production, but rather, an end in itself–for fun?

Here, we present findings from our research study conducted on a series of STEM workshops we developed and implemented in a local summer program for young Black boys. Data were collected and analyzed from surveys, student work, and field observations. Our preliminary findings suggest that youth began to acknowledge benefits in having both tech and sports within the same learning space. Specifically, they began to connect digital technologies with sports and consider them as mediating tools for acquiring and refining athletics skills.

Research on bridging sports and computer science education tends to center computing as primary, and sports as secondary. In contrast, this project aims to center sports as primary while introducing technology and computing as avenues to engage with sports more deeply. We collaborated with basketball coaches to design and implement campamento:bit, a summer program with a basketball team of Latinx students in Puerto Rico. This program integrated computing artifacts in a traditional basketball camp. We present insights from our experience testing this program with 11 participants over 5 days. We present select case studies of participant experiences and developing understandings using a qualitative-methods approach. These analyses present different shifts in athletes’ perceptions of computing and instantiated students’ interests in future possibilities involving computing. Furthermore, the analyses suggest strategies for organizing such technology-integrated experiences for youth in sports contexts.

Around the world, many K-12 school systems are seeking ways to provide youth with computer science (CS) learning experiences. Often organizations aim to develop these opportunities by building capacity among science, technology, engineering, and mathematics teachers. In other instances, school may engage with language arts, history, and library teachers to teach computer science content. Seldom, however, do schools leverage the rich opportunities for integrating computer science with physical education (PE). This paper explores an on-going partnership among university researchers, and elementary school coding and PE teachers. During spring of 2021, the group designed and tested coding and physical movement related activities for students to complete across their PE and coding classes. The team iterated on those activities throughout 2021 and 2022. This paper highlights the utility of this unique collaboration and describes some of the initial designs that emerged. The paper also touches on preliminary evaluation of the activities, and notes some of the project team's plans for future iterations. Broadly speaking, the activities piqued student interest and helped advance new perspectives of themselves, CS, and their teachers.

Programming can be an emotional experience, particularly for undergraduate students who are new to computer science. While researchers have interviewed novice programmers about their emotional experiences, it can be difficult to pinpoint the specific emotions that occur during a programming session. In this paper, we argue that electrodermal activity (EDA) sensors, which measure the physiological changes that are indicative of an emotional reaction, can provide a valuable new data source to help study student experiences. We conducted a study with 14 undergraduate students in which we collected EDA data while they worked on a programming problem. This data was then used to cue the participants’ recollections of their emotions during a retrospective interview about the programming experience. Using this methodology, we identified 21 distinct events that triggered student emotions, such as feeling anxiety due to a lack of perceived progress on the problem. We also identified common patterns in EDA data across multiple participants, such as a drop in their physiological reaction after developing a plan, corresponding with a calmer emotional state. These findings provide new information about how students experience programming that can inform research and practice, and also contribute initial evidence of the value of EDA data in supporting studies of emotions while programming.

In recent years, researchers have focused on the design and implementation of maker activities across formal and informal settings. As a result, the research community is gradually articulating the challenges and design considerations relating to these settings. These include: tools, facilitation, and curricular requirements. In this paper we present the design and implementation of Tinkering with Music, a 10-week youth club curriculum around popular music appreciation and instrument building with electronics. Reflecting on our design and implementation, we report on: (1) our curricular activities; (2) design challenges which we had to overcome throughout implementation, and (3) a failure to engender long term engagement with tools and practices from the curriculum.

Intergenerational making activities provide an opportunity for family collaboration where parents and children learn together. We discuss the facilitative moves that emerged between researchers, parents, and children during a half-day making program where participants played and created games. Four families with a variety of knowledge of digital fabrication technologies participated in three activities: playing a variety of games, designing and making their own games using arts and crafts materials, and optionally utilizing digital fabrication tools to complete their games. We position traditional fabrication and digital fabrication as two different modalities of making. Accordingly, we examine the facilitative moves and behavioral shifts that emerge across the two modalities and as observed through qualitative analysis. This work contributes insights to the field on program structure and the ways formal facilitators and parents can sustain child engagement in a making workshop.

Instructors are often multitasking in the classroom. This makes it increasingly difficult for them to pay attention to each individual’s engagement especially during activities where students are working in groups. In this paper, we describe a system that aids instructors in supporting group collaboration by utilizing a centralized, easy-to-navigate dashboard connected to multiple pods dispersed among groups of students in a classroom or laboratory. This allows instructors to check multiple qualities of the discussion such as: the usage of instructor specified keywords, relative participation of each individual, the speech acts students are using and different emotional characteristics of group language.

With millions of players worldwide, Minecraft has become a rich context for playing, socializing and learning for children. However, as is the case with many video games, players must rely heavily on vision to navigate and participate in the game. We present our Work-In-Progress on Tangicraft, a multimodal interface designed to empower visually impaired children to play and collaborate around Minecraft. Our work includes two strands of prototypes. The first is a haptic sensing wearable. The second is a set of tangible blocks that communicate with the game environment using webcam-enabled codes.

Video analysis is a staple of the education research community. For many contemporary education researchers, participation in the video coding process serves as a rite of passage. However, recent developments in multimodal learning analytics may help to accelerate and enhance this process by providing researchers with a more nuanced glimpse into a set of learning experiences. As an example of how to use multimodal learning analytics towards these ends, this paper includes a preliminary analysis from 54 college students, who completed two engineering design tasks in pairs. Gesture, speech and electro-dermal activation data were collected as students completed these tasks. The gesture data was used to learn a set of canonical clusters (N=4). A decision tree was trained based on individual students' cluster frequencies, and pre-post learning gains. The nodes in the decision tree were then used to identify a subset of video segments that were human coded based on prior work in learning analytics and engineering design. The combination of machine learning and human inference helps elucidate the practices that seem to correlate with student learning. In particular, both engagement and disengagement seem to correlate with student learning, albeit in a somewhat nuanced fashion.

Multimodal Learning Analytics (MMLA) captures, integrates and analyzes learning traces from different sources in order to obtain a more holistic understanding of the learning process, wherever it happens. MMLA leverages the increasingly widespread availability of diverse sensors, high-frequency data collection technologies and sophisticated machine learning and artificial intelligence techniques. The aim of this workshop is twofold: first, to expose participants to, and develop, different multimodal datasets that reflect how MMLA can bring new insights and opportunities to investigate complex learning processes and environments; second, to collaboratively identify a set of grand challenges for further MMLA research, built upon the foundations of previous workshops on the topic.

This is a proposal for organizing a Multimodal Learning Analytics (MLA) data challenge as part of the workshop offering of the Learning Analytics and Knowledge (LAK) conference. It explains the motivation of the event, its objectives, target groups, expected format, organization, dissemination strategy and schedule.

Multimodality is an integral part of teaching and learning. Over the past few decades researchers have been designing, creating and analyzing novel environments that enable students to experience and demonstrate learning through a variety of modalities. The recent availability of low cost multimodal sensors, advances in artificial intelligence and improved techniques for large scale data analysis have enabled researchers and practitioners to push the boundaries on multimodal learning and multimodal learning analytics. In an effort to continue these developments, the 2015 Multimodal Learning and Analytics Grand Challenge includes a combined focus on new techniques to capture multimodal learning data, as well as the development of rich, multimodal learning applications.

Multimodal analysis has long been an integral part of studying learning. Historically multimodal analyses of learning have been extremely laborious and time intensive. However, researchers have recently been exploring ways to use multimodal computational analysis in the service of studying how people learn in complex learning environments. In an effort to advance this research agenda, we present a comparative analysis of four different data segmentation techniques. In particular, we propose affect- and pose-based data segmentation, as alternatives to human-based segmentation, and fixed-window segmentation. In a study of ten dyads working on an open-ended engineering design task, we find that affect- and pose-based segmentation are more effective, than traditional approaches, for drawing correlations between learning-relevant constructs, and multimodal behaviors. We also find that pose-based segmentation outperforms the two more traditional segmentation strategies for predicting student success on the hands-on task. In this paper we discuss the algorithms used, our results, and the implications that this work may have in non-education-related contexts.

Multimodal analysis has had demonstrated effectiveness in studying and modeling several human-human and human-computer interactions. In this paper, we explore the role of multimodal analysis in the service of studying complex learning environments. We compare uni-modal and multimodal; manual and semi-automated methods for examining how students learn in a hands-on, engineering design context. Specifically, we compare human annotations, speech, gesture and electro-dermal activation data from a study (N=20) where student participating in two different experimental conditions. The experimental conditions have already been shown to be associated with differences in learning gains and design quality. Hence, one objective of this paper is to identify the behavioral practices that differed between the two experimental conditions, as this may help us better understand how the learning interventions work. An additional objective is to provide examples of how to conduct learning analytics research in complex environments and compare how the same algorithm, when used with different forms of data can provide complementary results.

Cognitive disequilibrium has received significant attention for its role in fostering student learning in intelligent tutoring systems and in complex learning environments. In this paper, we both add to and extend this discussion by analyzing the emergence of four affective states associated with disequilibrium: joy, surprise, neutrality and confusion; in a collaborative hands-on, engineering design task. Specifically, we conduct a comparison between two learning strategies to make salient how the strategies are associated with different affective states. This comparison is grounded in the construction of a probabilistic model of student affective state as defined by the frequency of each state, and the rate of transition between affective states. Through this comparison we confirm prior research that highlights the importance of confusion as a marker of knowledge construction, but put to question the notion that surprise is a significant mediator of cognitive disequilibrium. Overall, we show how modeling learner affect is useful for understanding and improving learning in complex, hands-on learning environments.

This paper summarizes the third Multimodal Learning Analytics Workshop and Grand Challenges (MLA'14). This subfield of Learning Analytics focuses on the interpretation of the multimodal interactions that occurs in learning environments, both digital and physical. This is a hybrid event that includes presentations about methods and techniques to analyze and merge the different signals captured from these environments (workshop session) and more concrete results from the application of Multimodal Learning Analytics techniques to predict the performance of students while solving math problems or presenting in the classroom (challenges sessions). A total of eight articles will be presented in this event. The main conclusion from this event is that Multimodal Learning Analytics is a desirable research endeavour that could produce results that can be currently applied to improve the learning process.

Advances in learning analytics are contributing new empirical findings, theories, methods, and metrics for understanding how students learn. It also contributes to improving pedagogical support for students' learning through assessment of new digital tools, teaching strategies, and curricula. Multimodal learning analytics (MMLA)[1] is an extension of learning analytics and emphasizes the analysis of natural rich modalities of communication across a variety of learning contexts. This MMLA Grand Challenge combines expertise from the learning sciences and machine learning in order to highlight the rich opportunities that exist at the intersection of these disciplines. As part of the Grand Challenge, researchers were asked to predict: (1) which student in a group was the dominant domain expert, and (2) which problems that the group worked on would be solved correctly or not. Analyses were based on a combination of speech, digital pen and video data. This paper describes the motivation for the grand challenge, the publicly available data resources and results reported by the challenge participants. The results demonstrate that multimodal prediction of the challenge goals: (1) is surprisingly reliable using rich multimodal data sources, (2) can be accomplished using any of the three modalities explored, and (3) need not be based on content analysis.

In this paper, we describe multimodal learning analytics techniques for understanding and identifying expertise as students engage in a hands-on building activity. Our techniques leverage process-oriented data, and demonstrate how this temporal data can be used to study student learning. The proposed techniques introduce useful insights in how to segment and analyze gesture- and action-based generally, and may also be useful for other sources of process rich data. Using this approach we uncover new ideas about how experts engage in building activities. Finally, a primary objective of this work is to motivate additional research and development in the area of authentic, automated, process-oriented assessments.

This summary describes the 1st International Workshop on Multimodal Learning Analytics. This area of study brings together the technologies of multimodal analysis with the learning sciences. The intersection of these domains should enable researchers to foster an improved understanding of student learning, lead to the creation of more natural and enriching learning interfaces, and motivate the development of novel techniques for tackling challenges that are specific of education.

Project-based learning has found its way into a range of formal and informal learning environments. However, systematically assessing these environments remains a significant challenge. Traditional assessments, which focus on learning outcomes, seem incongruent with the process-oriented goals of project-based learning. Multimodal interfaces and multimodal learning analytics hold significant promise for assessing learning in open-ended learning environments. With its rich integration of a multitude of data streams and naturalistic interfaces, this area of research may help usher in a new wave of education reform by supporting alternative modes of learning.

In this paper, we present an application framework for enabling education practitioners and researchers to develop interactive, multi-modal applications. These applications can be designed using typical HTML programming, and will enable a larger audience to make applications that incorporate speech recognition, gesture recognition and engagement detection. The application framework uses open-source software and inexpensive hardware that supports both multi-touch and multi-user capabilities.

Through the growing popularity of voice-enabled search, multimodal applications are finally starting to get into the hands of consumers. However, these applications are principally for mobile platforms and generally involve highly-moded interaction where the user has to click or hold a button in order to speak. Significant technical challenges remain in bringing multimodal interaction to other environments such as smart living rooms and classrooms, where users speech and gesture is directed toward large displays or interactive kiosks and the microphone and other sensors are `always on'. In this demonstration, we present a framework combining low cost hardware and open source software that lowers the barrier of entry for exploration of multimodal interaction in smart environments. Specifically, we will demonstrate the combination of infrared tracking, face detection, and open microphone speech recognition for media search (magicTV) and map navigation (magicMap).