Timely and high-quality feedback is essential for effective learning in programming courses; yet, providing such support at scale remains a challenge. While AI-based systems offer scalable and immediate help, their responses can occasionally be inaccurate or insufficient. Human instructors, in contrast, may bring more valuable expertise but are limited in time and availability. To address these limitations, we present a hybrid help framework that integrates AI-generated hints with an escalation mechanism, allowing students to request feedback from instructors when AI support falls short. This design leverages the strengths of AI for scale and responsiveness while reserving instructor effort for moments of greatest need. We deployed this tool in a data science programming course with 82 students. We observe that out of the total 673 AI-generated hints, students rated 146 (22%) as unhelpful. Among those, only 16 (11%) of the cases were escalated to the instructors. A qualitative investigation of instructor responses showed that those feedback instances were incorrect or insufficient roughly half of the time. This finding suggests that when AI support fails, even instructors with expertise may need to pay greater attention to avoid making mistakes. We will publicly release the tool for broader adoption and enable further studies in other classrooms. Our work contributes a practical approach to scaling high-quality support and informs future efforts to effectively integrate AI and humans in education.

Generative AI has the potential to enhance education by providing personalized feedback to students at scale. Recent work has proposed techniques to improve AI-generated programming hints and has evaluated their performance based on expert-designed rubrics or student ratings. However, it remains unclear how the rubrics used to design these techniques align with students' perceived helpfulness of hints. In this paper, we systematically study the mismatches in perceived hint quality from students' and experts' perspectives based on the deployment of AI-generated hints in a Python programming course. We analyze scenarios with discrepancies between student and expert evaluations, in particular, where experts rated a hint as high-quality while the student found it unhelpful. We identify key reasons for these discrepancies and classify them into categories, such as hints not accounting for the student's main concern or not considering previous help requests. Finally, we propose and discuss preliminary results on potential methods to bridge these gaps, first by extending the expert-designed quality rubric and then by adapting the hint generation process, e.g., incorporating the student's comments or history. These efforts contribute toward scalable, personalized, and pedagogically sound AI-assisted feedback systems, which are particularly important for high-enrollment educational settings.

Data Science (DS) has emerged as a new academic discipline where students are introduced to data-centric thinking and generating data-driven insights through programming. Unlike traditional introductory Computer Science (CS) education, which focuses on program syntax and core CS topics (e.g., algorithms and data structures), introductory DS education emphasizes skills such as analyzing data to gain insights by making effective use of programming libraries (e.g., re, NumPy, pandas, scikit-learn). To better understand learners' needs and pain points when they are introduced to DS programming, we investigated a large online course on data manipulation designed for graduate students who do not have a CS or Statistics undergraduate degree. We qualitatively analyzed students' incorrect code submissions for computational notebook-based assignments in Python. We identified common mistakes and grouped them into the following themes: (1) programming language and environment misconceptions, (2) logical mistakes due to data or problem-statement misunderstanding or incorrectly dealing with missing values, (3) semantic mistakes due to incorrect use of DS libraries, and (4) suboptimal coding. Our work provides instructors insights to understand student needs in introductory DS courses and improve course pedagogy, and recommendations for developing assessment and feedback tools to support students in large courses.

While learning analytics researchers have been diligently integrating trace log data into their studies, learners’ achievement goals are still predominantly measured by self-reported surveys. This study investigated the properties of trace data and survey data as representations of achievement goals. Through the lens of goal complex theory, we generated achievement goal clusters using latent variable mixture modeling applied to each kind of data. Findings show significant misalignment between these two data sources. Self-reported goals stated before learning do not translate into goal-relevant behaviors tracked using trace data collected during learning activities. While learners generally articulate an orientation towards mastery learning in self-report surveys, behavioral trace data showed a higher incidence of less engaged learning activities. These findings call into question the utility of survey-based measures when up-to-date achievement goal data are needed. Our results advance methodological and theoretical understandings of achievement goals in the modern age of learning analytics.

Despite a recent shift towards online learning, recommendations for multimedia design principles in programming-based instruction remain unclear. Specifically, how can we teach people to code, a text-heavy medium, properly in online instruction? This question is especially important since the text-based format of screencasts may interact with psychological mechanisms known to affect cognitive processing and learning. We investigate this question, and find that previous results from other domains do not necessarily hold in the programming education. We also explore how design changes in textual aids affect learners' performance in programming-based multimedia learning. Our results suggest that the redundancy effect does not significantly hinder learning, which conflicts with previous findings, and that the spatial contiguity effect occurs even between textual components. This work contributes to an evidence-based understanding of how to design more effective multimedia learning environments for programming-based instruction.

There has been limited research on how perceptions of socioeconomic status (SES) and opinion difference could influence peer feedback in Massive Open Online Courses (MOOCs). Using social comparison theory [12], we investigated the influence of ability and opinion-related factors on peer feedback text in a data science MOOC. Perceived SES of peers and the formality of written responses were used as the ability-related factor, while agreement between learners represented the opinion-related factor. We focused on understanding the behaviors of those learners who are most prevalent in MOOCs; those from high socioeconomic countries. Through two studies, we found a strong and repeated influence of agreement on affect and formality in feedback to peers. While a mediation effect of perceived SES was found, a significant effect of formality was not. This work contributes to an understanding of how social comparison theory can be operationalized in online peer writing environments.

There has been long-standing stereotypes on men and women's communication styles, such as men using more assertive or aggressive language and women showing more agreeableness and emotions in interactions. In the context of collaborative learning, male learners often believed to be more active participants while female learners are less engaged. To further explore gender differences in learners communication behavior and whether it has changed in the context of online synchronous collaboration, we examined students interactions at a sociocognitive level with a methodology called Group Communication Analysis (GCA). We found that there were no significant differences between men and women in the degree of participation. However, women exhibited significantly higher average social impact, responsivity and internal cohesion compared to men. We also compared the proportion of learners interaction profiles, and results suggest that women are more likely to be effective and cohesive communicators. We discussed implications of these findings for pedagogical practices to promote inclusivity and equity in collaborative learning online.

MOOCs have developed into multiple learning design models with a wide range of objectives. Teach-Outs are one such example, aiming to drive meaningful discussions around topics of pressing social urgency without the use of formal assessments. Given this approach, it is crucial to evaluate learners' engagement in the discussion forum to understand their experiences. This paper presents a pilot study that applied unsupervised natural language processing techniques to understand what and how students engage in dialogue in a Teach-Out. We used topic modeling to discover the emerging topics in the discussion forums and evaluated the on-topicness of the discussions (i.e. the degree to which discussions were relevant to the Teach-Out content). We also applied content analysis to investigate the sentiments associated with the discussions. We have taken a step toward extracting structure from students' discussions to understand learning behaviors happen in the discussion forum. This is the first study to analyze discussion forums in a Teach-Out.

In this work we investigate the use of deep learning for text analysis to measure elements of student thinking related to issues of privilege, oppression, diversity and social justice. We leverage historical expert annotations as well as a large lexical model to create a more generalizable vocabulary for identifying these characteristics in short student writing. We demonstrate the feasibility of this approach, and identify further areas for research.