Dialogue systems have long supported learner reflections, with theoretically grounded, rule-based designs offering structured scaffolding but often struggling to respond to shifts in engagement. Large Language Models (LLMs), in contrast, can generate context-sensitive responses but are not informed by decades of research on how learning interactions should be structured, raising questions about their alignment with pedagogical theories. This paper presents a hybrid dialogue system that embeds LLM responsiveness within a theory-aligned, rule-based framework to support learner reflections in a culturally responsive robotics summer camp. The rule-based structure grounds dialogue in self-regulated learning theory, while the LLM decides when and how to prompt deeper reflections, responding to evolving conversation context. We analyze themes across dialogues to explore how our hybrid system shaped learner reflections. Our findings indicate that LLM-embedded dialogues supported richer learner reflections on goals and activities, but also introduced challenges due to repetitiveness and misalignment in prompts, reducing engagement.

Socio-cultural context influences how learners engage with, experience, and make sense of learning. As such, Learning Analytics systems must intentionally account for this context in their design and interpretation. While many researchers in the field are already addressing these issues, sometimes implicitly or under different frameworks, we argue for a more unified focus on socio-cultural context as a guiding construct. Drawing on scholarship from education, sociology, anthropology, and psychology, we define socio-cultural context in terms of power, norms, and identity. Through two case studies of our own work on collaboration and engagement analytics, we illustrate how overlooking socio-cultural context can lead to incomplete systems that do not account for the variety of ways students might show up in learning spaces. We build on existing work and propose socio-cultural context as an organizing lens for future research. Doing so enables interdisciplinary collaboration and design that ultimately ensures analytics better serves learners.

Collaborative problem solving (CPS) is viewed as an essential 21st century skill for the modern workforce. Accordingly, researchers have been investigating how to conceptualize, assess, and develop pedagogical approaches to improve CPS. These efforts require theoretically-grounded and empirically-validated frameworks of CPS which have been emerging over the past decade with various levels of validity data. The present paper focuses on validating the generalized competency model (GCM) of CPS with respect to predicting individual learning outcomes following CPS among triads. The GCM consists of three main facets–constructing shared knowledge, negotiation/coordination, and maintaining team function–mapped to behavioral indicators (i.e., observable evidence). It hypothesizes that scores on all three facets should positively predict CPS outcomes, including group-to-individual learning transfer. We tested this hypothesis in a study where 249 students who comprised 83 triads engaged in collaborative gameplay with the Physics Playground game environment remotely via videoconferencing. We found that the only CPS facet predicting individual physics learning was maintaining team function, after accounting for pretest scores, students’ perceptions of team collaboration, and their perceived physics self-efficacy. This facet was also the only significant predictor of individual learning regardless of how facet scores were computed (i.e., reverse coding of negative indicators, separating the sums of positive and negative indicators, and no reverse coding of negative indicators). Implications for the GCM and other CPS frameworks are discussed.

Teaching is one of many professions for which personalized feedback and reflection can help improve dialogue and discussion between the professional and those they serve. However, professional development (PD) is often impersonal as human observation is labor-intensive. Data-driven PD tools in teaching are of growing interest, but open questions about how professionals engage with their data in practice remain. In this paper, we present ClassInSight, a tool that visualizes three levels of teachers’ discussion data and structures reflection. Through 22 reflection sessions and interviews with 5 high school science teachers, we found themes related to dissonance, contextualization, and sustainability in how teachers engaged with their data in the tool and in how their professional vision, the use of professional expertise to interpret events, shifted over time. We discuss guidelines for these conversational support tools to support personalized PD in professions beyond teaching where conversation and interaction are important.

The intersection of language and identity refers to the ways in which language use reflects and shapes one's sense of self and group membership. It is important to design voice-based technologies that represent and support diverse types of identities to increase accessibility. Due to constraints in the data and algorithms used to train and improve artificial intelligence (AI) systems like social robots or voice assistants, they are often unable to represent certain dialects of English and other languages. This could result in data bias, identity exclusion, and communication barriers that can impact the user's outcomes when using such dialects. Technology companies are discussing how to close this gap by improving training data; however, some researchers believe that the intersection of identity and technology calls for effective collaborative design methodologies that involve intended users. This work-in-progress paper presents preliminary findings on a participatory design method, comic-boarding, that can accomplish the above goal. Using comic-boarding, our intention is to create a space for critical reflection on language and identity in human-human and human-agent dialogue for children from marginalized communities.

Robot technologies have been introduced to computing education to engage learners. This study introduces the concept of co-creation with a robot agent into culturally-responsive computing (CRC). Co-creation with computer agents has previously focused on creating external artifacts. Our work differs by making the robot agent itself the co-created product. Through participatory design activities, we positioned adolescent girls and an agentic social robot as co-creators of the robot’s identity. Taking a thematic analysis approach, we examined how girls embody the role of creator and co-creator in this space. We identified themes surrounding who has the power to make decisions, what decisions are made, and how to maintain social relationship. Our findings suggest that co-creation with robot technology is a promising implementation vehicle for realizing CRC.

Artificial intelligence (AI) literacy is especially important for those who may not be well-represented in technology design. We worked with ten Black girls in fifth and sixth grade from a predominantly Black school to understand their perceptions around fair and accountable AI and how they can have an empowered role in the creation of AI. Thematic analysis of discussions and activity artifacts from a summer camp and after-school session revealed a number of findings around how Black girls: perceive AI, primarily consider fairness as niceness and equality (but may need support considering other notions, such as equity), consider accountability, and envision a just future. We also discuss how the learners can be positioned as decision-making designers in creating AI technology, as well as how AI literacy learning experiences can be empowering.

Computing education is important for K-12 learners, but not all learners resonate with common educational practices. Culturally responsive computing initiatives center and empower learners from diverse and historically excluded backgrounds. Recently, a number of educational programs have been developed and curated for an online experience. In this paper, we describe an online synchronous culturally responsive computing (CRC) camp for middle school girls (ages 11-14 years old) and report on challenges and successes from running the camp curriculum four times over the course of a year. We also describe core iterative changes we made between our runs. We then discuss lessons learned related to building rapport and connection among learners, centering learners of different backgrounds in an online synchronous environment, and facilitating reflection on power and identity aimed at positioning learners as techno-social change agents. Lastly, we offer recommendations for running online CRC experiences.

We investigated the generalizability of language-based analytics models across two collaborative problem solving (CPS) tasks: an educational physics game and a block programming challenge. We analyzed a dataset of 95 triads (N=285) who used videoconferencing to collaborate on both tasks for an hour. We trained supervised natural language processing classifiers on automatic speech recognition transcripts to predict the human-coded CPS facets (skills) of constructing shared knowledge, negotiation / coordination, and maintaining team function. We tested three methods for representing collaborative discourse: (1) deep transfer learning (using BERT), (2) n-grams (counts of words/phrases), and (3) word categories (using the Linguistic Inquiry Word Count [LIWC] dictionary). We found that the BERT and LIWC methods generalized across tasks with only a small degradation in performance (Transfer Ratio of .93 with 1 indicating perfect transfer), while the n-grams had limited generalizability (Transfer Ratio of .86), suggesting overfitting to task-specific language. We discuss the implications of our findings for deploying language-based collaboration analytics in authentic educational environments.

Computer science (CS) education is an important subject for K-12 students in an increasingly computational world. However, common CS education practices may not be inclusive of all learners. Culturally responsive computing (CRC) initiatives aim to center and empower learners from diverse and historically excluded backgrounds. With a sudden shift to online learning, virtual educational experiences have been developed. We describe three main findings from running three iterations of an online synchronous CRC camp for middle school girls, which are: (1) Integration of power, identity, and CS concepts, (2) Participation in CS vs. power and identity activities based on learners' backgrounds, and (3) Adjusting instructor expectations about learner engagement to be more open-ended.

Eye movements provide a window into cognitive processes, but much of the research harnessing this data has been confined to the laboratory. We address whether eye gaze can be passively, reliably, and privately recorded in real-world environments across extended timeframes using commercial-off-the-shelf (COTS) sensors. We recorded eye gaze data from a COTS tracker embedded in participants (N=20) work environments at pseudorandom intervals across a two-week period. We found that valid samples were recorded approximately 30% of the time despite calibrating the eye tracker only once and without placing any other restrictions on participants. The number of valid samples decreased over days with the degree of decrease dependent on contextual variables (i.e., frequency of video conferencing) and individual difference attributes (e.g., sleep quality and multitasking ability). Participants reported that sensors did not change or impact their work. Our findings suggest the potential for the collection of eye-gaze in authentic environments.

Culturally responsive STEM and computing initiatives aim to engage and embolden a diverse range of learners, center their identity and experiences in curriculum, and connect learners to each other and their communities. With an abrupt pivot to online learning at the beginning of 2020, more educational experiences have taken place virtually. We ran a virtual synchronous culturally responsive computing camp and saw that establishing the right environment online to support a good sense of connectedness was challenging. To investigate this further, we interviewed eight K-12 instructors of culturally responsive STEM and computing programs. Three themes emerged on defining and cultivating connectedness in learning experiences, the role of equity in supporting community online, and affordances of being online specific to culturally responsive perspectives. We support our thematic findings with vignettes from the camp data. In this study, we address K-12 culturally responsive STEM and computing instructors' beliefs, experiences, and approaches regarding cultivating connectedness online. This work fills a gap in understanding instructor perspectives on building in-program and broader community connections online from a culturally responsive STEM and computing lens.

Dialog with a social pedagogical robot or agent is a powerful way for kids to learn [1, 5] but may limit the formation of an agentic relationship with the technology [9]. One main purpose of conversational agents is to allow the user to have a natural interaction that reduces the need to learn artificial conventions [6], but dialog systems fall short with respect to failure recovery, vocabulary diversity, remembering conversational history, and other measures [2, 3]. Further, Hill et. al. [4] found that people adapt their model of communication to match a chatbot’s in the same way they do with a child or non-native speaker. Thus, users conversing with a pedagogical agent are implicitly trained to shape their behavior to suit the technology rather than shaping the technology. For young learners, particularly among populations that have been historically excluded from technology fields, this limits agency and reinforces marginalizing power structures [9].

In an increasingly globalized and service-oriented economy, people need to engage in computer-mediated collaborative problem solving (CPS) with diverse teams. However, teams routinely fail to live up to expectations, showcasing the need for technologies that help develop effective collaboration skills. We take a step in this direction by investigating how different dimensions of team diversity (demographic, personality, attitudes towards teamwork, prior domain experience) predict objective (e.g. effective solutions) and subjective (e.g. positive perceptions) collaborative outcomes. We collected data from 96 triads who engaged in a 30-minute CPS task via videoconferencing. We found that demographic diversity and differing attitudes towards teamwork predicted impressions of positive engagement, while personality diversity predicted learning outcomes. Importantly, these relationships were maintained after accounting for team makeup. None of the diversity measures predicted task performance. We discuss how our findings can be incorporated into technologies that aim to help diverse teams develop CPS skills.

Modeling team phenomena from multiparty interactions inherently requires combining signals from multiple teammates, often by weighting strategies. Here, we explored the hypothesis that strategic weighting signals from individual teammates would outperform an equal weighting baseline. Accordingly, we explored role-, trait-, and behavior-based weighting of behavioral signals across team members. We analyzed data from 101 triads engaged in computer-mediated collaborative problem solving (CPS) in an educational physics game. We investigated the accuracy of machine-learned models trained on facial expressions, acoustic-prosodics, eye gaze, and task context information, computed one-minute prior to the end of a game level, at predicting success at solving that level. AUROCs for unimodal models that equally weighted features from the three teammates ranged from .54 to .67, whereas a combination of gaze, face, and task context features, achieved an AUROC of .73. The various multiparty weighting strategies did not outperform an equal-weighting baseline. However, our best nonverbal model (AUROC = .73) outperformed a language-based model (AUROC = .67), and there were some advantages to combining the two (AUROC = .75). Finally, models aimed at prospectively predicting performance on a minute-by-minute basis from the start of the level achieved a lower, but still above-chance, AUROC of .60. We discuss implications for multiparty modeling of team performance and other team constructs.

Collaborative problem solving (CPS) in virtual environments is an increasingly important context of 21st century learning. However, our understanding of this complex and dynamic phenomenon is still limited. Here, we examine unimodal primitives (activity on the screen, speech, and body movements), and their multimodal combinations during remote CPS. We analyze two datasets where 116 triads collaboratively engaged in a challenging visual programming task using video conferencing software. We investigate how UI-interactions, behavioral primitives, and multimodal patterns were associated with teams' subjective and objective performance outcomes. We found that idling with limited speech (i.e., silence or backchannel feedback only) and without movement was negatively correlated with task performance and with participants' subjective perceptions of the collaboration. However, being silent and focused during solution execution was positively correlated with task performance. Results illustrate that in some cases, multimodal patterns improved the predictions and improved explanatory power over the unimodal primitives. We discuss how the findings can inform the design of real-time interventions for remote CPS.

Multiparty collaborative problem solving - an increasingly important context in the 21st century workforce - suffers from a degradation of social and behavioral signals when attempted remotely, resulting in suboptimal outcomes. We investigate teams' multidimensional patterns of visual attention during a collaborative problem-solving task with an eye for leveraging insights to improve collaborative interfaces. Fifty-seven novices (forming 19 triads) engaged in a challenging programming task (Minecraft Hour of Code) using videoconferencing software with screen sharing. To discover patterns of individual-level gaze-UI coupling(coordination of a teammate's attention with respect to changes in the user interface) and team-level gaze-UI regularity (dynamics of teams' collective attention in context with changes in the user interface), we applied cross- and multidimensional recurrence quantification analyses, respectively. Individuals' eye gaze was significantly coupled with the ongoing screen activity whereas teams displayed significant patterns of gaze regularity, suggesting repetitive patterns in teams' attention. These measures predicted expert-coded collaborative processes of constructing shared knowledge and negotiation and coordination (but not maintaining team function) and correlated with task score (r = .425). They also predicted individually assessed subjective perceptions of team performance and the collaboration process, but not individual's learning or team's task scores. We discuss implications of our findings for the design of intelligent collaborative interfaces.

We adopt a multimodal approach to investigating team interactions in the context of remote collaborative problem solving (CPS). Our goal is to understand multimodal patterns that emerge and their relation with collaborative outcomes. We measured speech rate, body movement, and galvanic skin response from 101 triads (303 participants) who used video conferencing software to collaboratively solve challenging levels in an educational physics game. We use multi-dimensional recurrence quantification analysis (MdRQA) to quantify patterns of team-level regularity, or repeated patterns of activity in these three modalities. We found that teams exhibit significant regularity above chance baselines. Regularity was unaffected by task factors. but had a quadratic relationship with session time in that it initially increased but then decreased as the session progressed. Importantly, teams that produce more varied behavioral patterns (irregularity) reported higher emotional valence and performed better on a subset of the problem solving tasks. Regularity did not predict arousal or subjective perceptions of the collaboration. We discuss implications of our findings for the design of systems that aim to improve collaborative outcomes by monitoring the ongoing collaboration and intervening accordingly.

Collaborative problem solving (CPS) is a crucial 21st century skill; however, current technologies fall short of effectively supporting CPS processes, especially for remote, computer-enabled interactions. In order to develop next-generation computer-supported collaborative systems that enhance CPS processes and outcomes by monitoring and responding to the unfolding collaboration, we investigate automated detection of three critical CPS process ? construction of shared knowledge, negotiation/coordination, and maintaining team function ? derived from a validated CPS framework. Our data consists of 32 triads who were tasked with collaboratively solving a challenging visual computer programming task for 20 minutes using commercial videoconferencing software. We used automatic speech recognition to generate transcripts of 11,163 utterances, which trained humans coded for evidence of the above three CPS processes using a set of behavioral indicators. We aimed to automate the trained human-raters' codes in a team-independent fashion (current study) in order to provide automatic real-time or offline feedback (future work). We used Random Forest classifiers trained on the words themselves (bag of n-grams) or with word categories (e.g., emotions, thinking styles, social constructs) from the Linguistic Inquiry Word Count (LIWC) tool. Despite imperfect automatic speech recognition, the n-gram models achieved AUROC (area under the receiver operating characteristic curve) scores of .85, .77, and .77 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively; these reflect 70%, 54%, and 54% improvements over chance. The LIWC-category models achieved similar scores of .82, .74, and .73 (64%, 48%, and 46% improvement over chance). Further, the LIWC model-derived scores predicted CPS outcomes more similar to human codes, demonstrating predictive validity. We discuss embedding our models in collaborative interfaces for assessment and dynamic intervention aimed at improving CPS outcomes.

We model coordination and coregulation patterns in 33 triads engaged in collaboratively solving a challenging computer programming task for approximately 20 minutes. Our goal is to prospectively model speech rate (words/sec) - an important signal of turn taking and active participation - of one teammate (A or B or C) from time lagged nonverbal signals (speech rate and acoustic-prosodic features) of the other two (i.e., A + B → C; A + C → B; B + C → A) and task-related context features. We trained feed-forward neural networks (FFNNs) and long short-term memory recurrent neural networks (LSTMs) using group-level nested cross-validation. LSTMs outperformed FFNNs and a chance baseline and could predict speech rate up to 6s into the future. A multimodal combination of speech rate, acoustic-prosodic, and task context features outperformed unimodal and bimodal signals. The extent to which the models could predict an individual's speech rate was positively related to that individual's scores on a subsequent posttest, suggesting a link between coordination/coregulation and collaborative learning outcomes. We discuss applications of the models for real-time systems that monitor the collaborative process and intervene to promote positive collaborative outcomes.

Mind wandering (MW) is a ubiquitous phenomenon in which attention involuntarily shifts from task-related processing to task-unrelated thoughts. This study reports preliminary results of a video-based MW detector during film viewing. We collected training data in a study where participants self-reported when they caught themselves MW over the course of watching a 32.5 minute commercial film. We trained classification models on automatically extracted facial features and bodily movement and were able to detect MW with an F1 of .30. The model was successful in reproducing the MW distribution obtained from the self-reports