Smart home technologies are often designed to meet the needs of adults, yet children and pets also live with these systems without being meaningfully considered in their design. Child-Computer Interaction (CCI) researchers have shown the value of studying children's experiences and ideation of technologies used in the domestic space. In this pictorial, we explore experiences at the intersection of children, pets, and smart home technologies by analyzing data from an in-home study with 6-to-11-year-olds. Our analysis identifies five themes in how children perceive smart home technologies in the context of pet care: convenience, presence, physical comfort, emotional wellbeing, and responsibility. Grounded in children's everyday routines of playing with and looking after their pets, this work offers design directions for domestic technologies that account for non-human household members.

Smart home technologies have become common in family homes, making even young children inevitable users of these technologies. However, these systems are typically designed for individual adults, creating family tensions and conflicts over children’s access, safety, and appropriate smart home use. To investigate children’s and parents’ individual and joint smart home needs and dynamics, we conducted an in-home study with nine families (children aged 6-11). We identify four key parent-child tensions with smart home technologies, including struggles over parental protection versus children’s autonomy, differing views on technology’s purpose, disagreements over technology-enforced routines, and children’s vulnerability to embedded commercialism. Our work reconceptualizes parental mediation as a process of “tension management” rather than the application of static rules. This research challenges the dominant individual-centric choice architecture in smart home design, calling for a family-centered approach that acknowledges and adapts to the fluid, complex, and negotiated reality of modern family life.

Timely and high-quality feedback is essential for effective learning in programming courses; yet, providing such support at scale remains a challenge. While AI-based systems offer scalable and immediate help, their responses can occasionally be inaccurate or insufficient. Human instructors, in contrast, may bring more valuable expertise but are limited in time and availability. To address these limitations, we present a hybrid help framework that integrates AI-generated hints with an escalation mechanism, allowing students to request feedback from instructors when AI support falls short. This design leverages the strengths of AI for scale and responsiveness while reserving instructor effort for moments of greatest need. We deployed this tool in a data science programming course with 82 students. We observe that out of the total 673 AI-generated hints, students rated 146 (22%) as unhelpful. Among those, only 16 (11%) of the cases were escalated to the instructors. A qualitative investigation of instructor responses showed that those feedback instances were incorrect or insufficient roughly half of the time. This finding suggests that when AI support fails, even instructors with expertise may need to pay greater attention to avoid making mistakes. We will publicly release the tool for broader adoption and enable further studies in other classrooms. Our work contributes a practical approach to scaling high-quality support and informs future efforts to effectively integrate AI and humans in education.

Peer Reviewed

Generative AI has the potential to enhance education by providing personalized feedback to students at scale. Recent work has proposed techniques to improve AI-generated programming hints and has evaluated their performance based on expert-designed rubrics or student ratings. However, it remains unclear how the rubrics used to design these techniques align with students' perceived helpfulness of hints. In this paper, we systematically study the mismatches in perceived hint quality from students' and experts' perspectives based on the deployment of AI-generated hints in a Python programming course. We analyze scenarios with discrepancies between student and expert evaluations, in particular, where experts rated a hint as high-quality while the student found it unhelpful. We identify key reasons for these discrepancies and classify them into categories, such as hints not accounting for the student's main concern or not considering previous help requests. Finally, we propose and discuss preliminary results on potential methods to bridge these gaps, first by extending the expert-designed quality rubric and then by adapting the hint generation process, e.g., incorporating the student's comments or history. These efforts contribute toward scalable, personalized, and pedagogically sound AI-assisted feedback systems, which are particularly important for high-enrollment educational settings.

Generative AI and large language models hold great promise in enhancing programming education by automatically generating individualized feedback for students. We investigate the role of generative AI models in providing human tutor-style programming hints to help students resolve errors in their buggy programs. Recent works have benchmarked state-of-the-art models for various feedback generation scenarios; however, their overall quality is still inferior to human tutors and not yet ready for real-world deployment. In this paper, we seek to push the limits of generative AI models toward providing high-quality programming hints and develop a novel technique, GPT4HINTS-GPT3.5VAL. As a first step, our technique leverages GPT-4 as a “tutor” model to generate hints – it boosts the generative quality by using symbolic information of failing test cases and fixes in prompts. As a next step, our technique leverages GPT-3.5, a weaker model, as a “student” model to further validate the hint quality – it performs an automatic quality validation by simulating the potential utility of providing this feedback. We show the efficacy of our technique via extensive evaluation using three real-world datasets of Python programs covering a variety of concepts ranging from basic algorithms to regular expressions and data analysis using pandas library.

This paper explores the space of optimizing feedback mechanisms in complex domains such as data science, by combining two prevailing approaches: Artificial Intelligence (AI) and learnersourcing. Towards addressing the challenges posed by each approach, this work compares traditional learnersourcing with an AI-supported approach. We report on the results of a randomized controlled experiment conducted with 72 Master’s level students in a data visualization course, comparing two conditions: students writing hints independently versus revising hints generated by GPT-4. The study aimed to evaluate the quality of learnersourced hints, examine the impact of student performance on hint quality, gauge learner preference for writing hints with versus without AI support, and explore the potential of the student-AI collaborative exercise in fostering critical thinking about LLMs. Based on our findings, we provide insights for designing learnersourcing activities leveraging AI support and optimizing students’ learning as they interact with LLMs.

In this study, we designed a tool to investigate the relationship between students' ability to render accurate judgements of learning (JOLs) with decision-making behavior when annotating their own work and comparing it with perceived AI-generated annotations. Our findings suggest that students rarely adjust their JOLs after seeing the AI annotations, indicative of a strong self-confirmation bias. Trust in the AI tool was associated with a decreased likelihood of changing initial judgments, in part due to the similarity of AI annotations with their own. The process of using the tool to self-annotate was found to enhance performance on a post-test. Emphasizing clear learning objectives and being transparent with the limitations of AI functions may improve the effectiveness of such tools as a way to provide quick feedback and mitigate hesitancy towards wider-scale adoption.

aendees to leave the workshop with a practical understanding of how to engage with learnersourcing.Participants will get hands-on experience with current tools,

Data Science (DS) has emerged as a new academic discipline where students are introduced to data-centric thinking and generating data-driven insights through programming. Unlike traditional introductory Computer Science (CS) education, which focuses on program syntax and core CS topics (e.g., algorithms and data structures), introductory DS education emphasizes skills such as analyzing data to gain insights by making effective use of programming libraries (e.g., re, NumPy, pandas, scikit-learn). To better understand learners' needs and pain points when they are introduced to DS programming, we investigated a large online course on data manipulation designed for graduate students who do not have a CS or Statistics undergraduate degree. We qualitatively analyzed students' incorrect code submissions for computational notebook-based assignments in Python. We identified common mistakes and grouped them into the following themes: (1) programming language and environment misconceptions, (2) logical mistakes due to data or problem-statement misunderstanding or incorrectly dealing with missing values, (3) semantic mistakes due to incorrect use of DS libraries, and (4) suboptimal coding. Our work provides instructors insights to understand student needs in introductory DS courses and improve course pedagogy, and recommendations for developing assessment and feedback tools to support students in large courses.

Smart home technologies are making their way into families. Parents' and children's shared use of smart home technologies has received growing attention in CSCW and related research communities. Families and children are also frequently featured as target audiences in smart home product marketing. However, there is limited knowledge of how exactly children and family interactions are portrayed in smart home product marketing, and to what extent those portrayals align with the actual consideration of children and families in product features and resources for child safety and privacy. We conducted a content analysis of product websites and online resources of 102 smart home products, as these materials constitute a main marketing channel and information source about products for consumers. We found that despite featuring children in smart home marketing, most analyzed product websites did not mention child safety features and lacked sufficient information on how children's data is collected and used. Specifically, our findings highlight misalignments in three aspects: (1) children are depicted as users of smart home products but there are insufficient child-friendly product features; (2) harmonious child-product co-presence is portrayed but potential child safety issues are neglected; and (3) children are shown as the subject of monitoring and datafication but there is limited information on child data collection and use. We discuss how parent-child relationships and parenting may be negatively impacted by such marketing depictions, and we provide design and policy recommendations for better incorporating child safety and privacy considerations into smart home products.

Multi-stage programming tutorials are key learning resources for programmers, using progressive incremental steps to teach them how to build larger software systems. A good multi-stage tutorial describes the code clearly, explains the rationale and code changes for each step, and allows readers to experiment as they work through the tutorial. In practice, it is time-consuming for authors to create tutorials with these attributes. In this paper, we introduce Colaroid, an interactive authoring tool for creating high quality multi-stage tutorials. Colaroid tutorials are augmented computational notebooks, where snippets and outputs represent a snapshot of a project, with source code differences highlighted, complete source code context for each snippet, and the ability to load and tinker with any stage of the project in a linked IDE. In two laboratory studies, we found Colaroid makes it easy to create multi-stage tutorials, while offering advantages to readers compared to video and web-based tutorials.

Regular expressions (regex) are a text processing method widely used in data analysis, web scraping, and input validation. However, students find regular expressions difficult to create since they use a terse language of characters. Parsons problems can be a more efficient way to practice programming than typing the equivalent code with similar learning gains. In traditional Parsons problems, learners place mixed-up fragments with one or more lines in each fragment in order to solve a problem. To investigate learning regex with Parsons problems, we introduce micro Parsons problems, in which learners assemble fragments in a single line. We conducted both a think-aloud study and a large-scale between-subjects field study to evaluate this new approach. The think-aloud study provided insights into learners' perceptions of the advantages and disadvantages of solving micro Parsons problems versus traditional text-entry problems, student preferences, and revealed design considerations for micro Parsons problems. The between-subjects field study of 3,752 participants compared micro Parsons problems with text-entry problems as an optional assignment in a MOOC. The dropout rate for the micro Parsons condition was significantly lower than the text-entry condition. No significant difference was found for the learning gain on questions testing comprehensive regex skills between the two conditions, but the micro Parsons group had a significantly higher learning gain on multiple choice questions which tested understanding of regex characters.

While learning analytics researchers have been diligently integrating trace log data into their studies, learners’ achievement goals are still predominantly measured by self-reported surveys. This study investigated the properties of trace data and survey data as representations of achievement goals. Through the lens of goal complex theory, we generated achievement goal clusters using latent variable mixture modeling applied to each kind of data. Findings show significant misalignment between these two data sources. Self-reported goals stated before learning do not translate into goal-relevant behaviors tracked using trace data collected during learning activities. While learners generally articulate an orientation towards mastery learning in self-report surveys, behavioral trace data showed a higher incidence of less engaged learning activities. These findings call into question the utility of survey-based measures when up-to-date achievement goal data are needed. Our results advance methodological and theoretical understandings of achievement goals in the modern age of learning analytics.

The first annual workshop on Learnersourcing: Student-generated Content @ Scale is taking place at Learning @ Scale 2022. This hybrid workshop will expose attendees to the ample opportunities in the learnersourcing space, including instructors, researchers, learning engineers, and many other roles. We believe participants from a wide range of backgrounds and prior knowledge on learnersourcing can both benefit and contribute to this workshop, as learnersourcing draws on work from education, crowdsourcing, learning analytics, data mining, ML/NLP, and many more fields. Additionally, as the learnersourcing process involves many stakeholders (students, instructors, researchers, instructional designers, etc.), multiple viewpoints can help to inform what future student-generated content might be useful, new and better ways to assess the quality of the content and spark potential collaboration efforts between attendees. We ultimately want to show how everyone can make use of learnersourcing and have participants gain hands-on experience using existing tools, create their own learnersourcing activities using them or their own platforms, and take part in discussing the next challenges and opportunities in the learnersourcing space. Our hope is to attract attendees interested in scaling the generation of instructional and assessment content and those interested in the use of online learning platforms.

Given the growing interest in learnersourcing -- a pedagogically supported form of crowdsourcing that harnesses the knowledge and creativity of learners for the creation of learning resources -- we propose a theoretical framework to study, design, and deploy learnersourcing systems. By integrating ideas from crowdsourcing and learning theories focusing on learner-centered pedagogy, this work provides a review and classification of learnersourcing systems from the perspective of its three groups of stakeholders: (1) contributors, who are the learners who contribute new learning artifacts, (2) beneficiaries, who are the learners who learn from these artifacts, and (3) the instructional team who design and deploy learnersourcing tasks. The framework serves as a heuristic device for designing new learnersourcing systems and for considering the broader implications for learnersourcing in terms of workflow design, incentivizing contributors, quality-control, learning outcomes, delivering personalized learning experiences, ethical considerations, and its complementary relationship with AI in education.

Despite a recent shift towards online learning, recommendations for multimedia design principles in programming-based instruction remain unclear. Specifically, how can we teach people to code, a text-heavy medium, properly in online instruction? This question is especially important since the text-based format of screencasts may interact with psychological mechanisms known to affect cognitive processing and learning. We investigate this question, and find that previous results from other domains do not necessarily hold in the programming education. We also explore how design changes in textual aids affect learners' performance in programming-based multimedia learning. Our results suggest that the redundancy effect does not significantly hinder learning, which conflicts with previous findings, and that the spatial contiguity effect occurs even between textual components. This work contributes to an evidence-based understanding of how to design more effective multimedia learning environments for programming-based instruction.

Question generation as a form of learnersourcing is both a metacognitive learning activity for students that encourages the development of higher-order thinking skills and a method for producing question banks and assessments. To better understand the motivations for learners who engage in learnersourcing and its impacts on student learning, we conducted an experiment that measured the effects of Multiple Choice Question (MCQ) generation in an introductory data science MOOC. We compared two approaches to question generation: (i) as a required activity, and (ii) as an optional activity. In both cases, the learnersourcing activity was part of the student summative evaluation. We found that learners value creating questions more, and create higher quality questions when they choose to do so compared to when it is required. At the same time there is a significant reduction in instructor evaluation workload in large-scale courses when learners engage by choice due to self-selection. Thus, we propose choice-based learnersourcing as a new form of scalable personalized learning design for MOOCs in particular. In addition, we contribute an exploration of the factors that influence learner choice to create (or not create) an MCQ, which can help contextualize the propensity of learners to engage in such learnersourcing activities.

Concerns about child physical and digital safety are emerging with families' adoption of smart home technologies such as robot vacuums and smart speakers. To better understand parents' definitions and perceptions of child safety regarding smart home technologies, we interviewed 23 parents who are smart home adopters. We contribute insights into parents' perceptions of the physical and digital safety risks smart home technologies pose to children, and how such perceptions formed and changed across three phases. In acquiring smart home devices, parents already considered whether the device could cause physical harm to their children or pose privacy and security risks. Once children become active users of smart home technologies, parents however reported encountering unanticipated physical safety risks and digital safety issues (e.g., exposure to unsuitable content) that required their mitigation strategies. As their children grow up, parents further expressed the need to shift attention from physical safety to digital safety. Parents' safety perceptions influence how they involve children in smart home interactions and implement mitigation strategies, such as restricting access to certain devices and using parental controls. We identify six factors that shape parents' perception and evaluation of smart home safety risks to children, including parenting style, parents' tech-savviness, parents' trust in tech companies, children's age and developmental differences, news media, and device characteristics. We provide design and policy recommendations to better protect children's safety in the smart home environment.

Peer assessment, as a form of collaborative learning, can engage students in active learning and improve their learning gains. However, current teaching platforms and programming environments provide little support to integrate peer assessment for in-class programming exercises. We identified challenges in conducting such exercises and adopting peer assessment through formative interviews with instructors of introductory programming courses. To address these challenges, we introduce PuzzleMe, a tool to help Computer Science instructors to conduct engaging in-class programming exercises. PuzzleMe leverages peer assessment to support a collaboration model where students provide timely feedback on their peers' work. We propose two assessment techniques tailored to in-class programming exercises: live peer testing and live peer code review. Live peer testing can improve students' code robustness by allowing them to create and share lightweight tests with peers. Live peer code review can improve code understanding by intelligently grouping students to maximize meaningful code reviews. A two-week deployment study revealed that PuzzleMe encourages students to write useful test cases, identify code problems, correct misunderstandings, and learn a diverse set of problem-solving approaches from peers.

When teams of data scientists collaborate on computational notebooks, their discussions often contain valuable insight into their design decisions. These discussions not only explain analysis in the current notebook but also alternative paths, which are often poorly documented. However, these discussions are disconnected from the notebooks for which they could provide valuable context. We propose Callisto, an extension to computational notebooks that captures and stores contextual links between discussion messages and notebook elements with minimal effort from users. Callisto allows notebook readers to better understand the current notebook content and the overall problem-solving process that led to it, by making it possible to browse the discussions and code history relevant to any part of the notebook. This is particularly helpful for onboarding new notebook collaborators to avoid misinterpretations and duplicated work, as we found in a two-stage evaluation with 32 data science students.

Large-scale online learning environments present new opportunities to address the need for greater inclusivity in education. Unlike residential environments, which have physical and logistic constraints (e.g., classroom configurations, sizes, and scheduling) that impede our ability to enact more inclusive pedagogy, online learning environments can be personalized and adapted to individual learner needs. As these environments are completely technology mediated, they offer an almost infinite design space for innovation. Social-scientific research on inclusivity in residential settings provides insight into how we might design for online learning environments, however evidence of efficacious digital implementations of these insights is limited. This workshop aims to advance our understanding of the ways in which adaptivity can be leveraged to buttress inclusivity in STEM learning. Through brief paper presentations and collaborative activities we intend to outline design opportunities in the scaled learning space for creating more inclusive environments.

Early warning dashboards in higher education analyze student data to enable early identification of underperforming students, allowing timely interventions by faculty and staff. To understand perceptions regarding the ethics and impact of such learning analytics applications, we conducted a multi-stakeholder analysis of an early-warning dashboard deployed at the University of Michigan through semi-structured interviews with the system's developers, academic advisors (the primary users), and students. We identify multiple tensions among and within the stakeholder groups, especially with regard to awareness, understanding, access and use of the system. Furthermore, ambiguity in data provenance and data quality result in differing levels of reliance and concerns about the system among academic advisors and students. While students see the system's benefits, they argue for more involvement, control, and informed consent regarding the use of student data. We discuss our findings' implications for the ethical design and deployment of learning analytics applications in higher education. Early warning dashboards in higher education analyze student data to enable early identification of underperforming students, allowing timely interventions by faculty and staff. To understand perceptions regarding the ethics and impact of such learning analytics applications, we conducted a multi-stakeholder analysis of an early-warning dashboard deployed at the University of Michigan through semi-structured interviews with the system's developers, academic advisors (the primary users), and students. We identify multiple tensions among and within the stakeholder groups, especially with regard to awareness, understanding, access, and use of the system. Furthermore, ambiguity in data provenance and data quality result in differing levels of reliance and concerns about the system among academic advisors and students. While students see the system's benefits, they argue for more involvement, control, and informed consent regarding the use of student data. We discuss our findings' implications for the ethical design and deployment of learning analytics applications in higher education.

There has been limited research on how perceptions of socioeconomic status (SES) and opinion difference could influence peer feedback in Massive Open Online Courses (MOOCs). Using social comparison theory [12], we investigated the influence of ability and opinion-related factors on peer feedback text in a data science MOOC. Perceived SES of peers and the formality of written responses were used as the ability-related factor, while agreement between learners represented the opinion-related factor. We focused on understanding the behaviors of those learners who are most prevalent in MOOCs; those from high socioeconomic countries. Through two studies, we found a strong and repeated influence of agreement on affect and formality in feedback to peers. While a mediation effect of perceived SES was found, a significant effect of formality was not. This work contributes to an understanding of how social comparison theory can be operationalized in online peer writing environments.

Predictive modeling has been a core area of learning analytics research over the past decade, with such models currently deployed in a variety of educational contexts from MOOCs to K-12. However, analyses of the differential effectiveness of these models across demographic, identity, or other groups has been scarce. In this paper, we present a method for evaluating unfairness in predictive student models. We define this in terms of differential accuracy between subgroups, and measure it using a new metric we term the Absolute Between-ROC Area (ABROCA). We demonstrate the proposed method through a gender-based "slicing analysis" using five different models replicated from other works and a dataset of 44 unique MOOCs and over four million learners. Our results demonstrate (1) significant differences in model fairness according to (a) statistical algorithm and (b) feature set used; (2) that the gender imbalance ratio, curricular area, and specific course used for a model all display significant association with the value of the ABROCA statistic; and (3) that there is not evidence of a strict tradeoff between performance and fairness. This work provides a framework for quantifying and understanding how predictive models might inadvertently privilege, or disparately impact, different student subgroups. Furthermore, our results suggest that learning analytics researchers and practitioners can use slicing analysis to improve model fairness without necessarily sacrificing performance.1

Privacy concerns may lead people to opt-in or opt-out of having their educational data collected. These decisions may impact the performance of educational predictive models. To understand this, we conducted a survey to determine the propensity of students to withhold or grant access to their data for the purposes of training predictive models. We simulated the effects of opt-out on the accuracy of educational predictive models by dropping a random sample of data over a range of increments, and then contextualize our findings using the survey results. We find that grade predictive models are fairly robust and that kappa scores do not decrease unless there is signiicant opt-out, but when there is, the deteriorating performance disproportionately affects certain subpopulations.

There has been long-standing stereotypes on men and women's communication styles, such as men using more assertive or aggressive language and women showing more agreeableness and emotions in interactions. In the context of collaborative learning, male learners often believed to be more active participants while female learners are less engaged. To further explore gender differences in learners communication behavior and whether it has changed in the context of online synchronous collaboration, we examined students interactions at a sociocognitive level with a methodology called Group Communication Analysis (GCA). We found that there were no significant differences between men and women in the degree of participation. However, women exhibited significantly higher average social impact, responsivity and internal cohesion compared to men. We also compared the proportion of learners interaction profiles, and results suggest that women are more likely to be effective and cohesive communicators. We discussed implications of these findings for pedagogical practices to promote inclusivity and equity in collaborative learning online.

Randomized experiments ensure robust causal inference that is critical to effective learning analytics research and practice. However, traditional randomized experiments, like A/B tests, are limiting in large scale digital learning environments. While traditional experiments can accurately compare two treatment options, they are less able to inform how to adapt interventions to continually meet learners' diverse needs. In this work, we introduce a trial design for developing adaptive interventions in scaled digital learning environments -- the sequential randomized trial (SRT). With the goal of improving learner experience and developing interventions that benefit all learners at all times, SRTs inform how to sequence, time, and personalize interventions. In this paper, we provide an overview of SRTs, and we illustrate the advantages they hold compared to traditional experiments. We describe a novel SRT run in a large scale data science MOOC. The trial results contextualize how learner engagement can be addressed through culturally-targeted reminder emails. We also provide practical advice for researchers who aim to run their own SRTs to develop adaptive interventions in scaled digital learning environments.

MOOCs have developed into multiple learning design models with a wide range of objectives. Teach-Outs are one such example, aiming to drive meaningful discussions around topics of pressing social urgency without the use of formal assessments. Given this approach, it is crucial to evaluate learners' engagement in the discussion forum to understand their experiences. This paper presents a pilot study that applied unsupervised natural language processing techniques to understand what and how students engage in dialogue in a Teach-Out. We used topic modeling to discover the emerging topics in the discussion forums and evaluated the on-topicness of the discussions (i.e. the degree to which discussions were relevant to the Teach-Out content). We also applied content analysis to investigate the sentiments associated with the discussions. We have taken a step toward extracting structure from students' discussions to understand learning behaviors happen in the discussion forum. This is the first study to analyze discussion forums in a Teach-Out.

Effective collaboration in data science can leverage domain expertise from each team member and thus improve the quality and efficiency of the work. Computational notebooks give data scientists a convenient interactive solution for sharing and keeping track of the data exploration process through a combination of code, narrative text, visualizations, and other rich media. In this paper, we report how synchronous editing in computational notebooks changes the way data scientists work together compared to working on individual notebooks. We first conducted a formative survey with 195 data scientists to understand their past experience with collaboration in the context of data science. Next, we carried out an observational study of 24 data scientists working in pairs remotely to solve a typical data science predictive modeling problem, working on either notebooks supported by synchronous groupware or individual notebooks in a collaborative setting. The study showed that working on the synchronous notebooks improves collaboration by creating a shared context, encouraging more exploration, and reducing communication costs. However, the current synchronous editing features may lead to unbalanced participation and activity interference without strategic coordination. The synchronous notebooks may also amplify the tension between quick exploration and clear explanations. Building on these findings, we propose several design implications aimed at better supporting collaborative editing in computational notebooks, and thus improving efficiency in teamwork among data scientists.

The field of learning analytics needs to adopt a more rigorous approach for predictive model evaluation that matches the complex practice of model-building. In this work, we present a procedure to statistically test hypotheses about model performance which goes beyond the state-of-the-practice in the community to analyze both algorithms and feature extraction methods from raw data. We apply this method to a series of algorithms and feature sets derived from a large sample of Massive Open Online Courses (MOOCs). While a complete comparison of all potential modeling approaches is beyond the scope of this paper, we show that this approach reveals a large gap in dropout prediction performance between forum-, assignment-, and clickstream-based feature extraction methods, where the latter is significantly better than the former two, which are in turn indistinguishable from one another. This work has methodological implications for evaluating predictive or AI-based models of student success, and practical implications for the design and targeting of at-risk student models and interventions.

Big data repositories from online learning platforms such as Massive Open Online Courses (MOOCs) represent an unprecedented opportunity to advance research on education at scale and impact a global population of learners. To date, such research has been hindered by poor reproducibility and a lack of replication, largely due to three types of barriers: experimental, inferential, and data. We present a novel system for large-scale computational research, the MOOC Replication Framework (MORF), to jointly address these barriers. We discuss MORF’s architecture, an open- source platform-as-a-service (PaaS) which includes a simple, flexible software API providing for multiple modes of research (predictive modeling or production rule analysis) integrated with a high-performance computing environment. All experiments conducted on MORF use executable Docker containers which ensure complete reproducibility while allowing for the use of any software or language which can be installed in the linux-based Docker container. Each experimental artifact is assigned a DOI and made publicly available. MORF has the potential to accelerate and democratize research on its massive data repository, which currently includes over 200 MOOCs, as demonstrated by initial research conducted on the platform. We also highlight ways in which MORF represents a solution template to a more general class of problems faced by computational researchers in other domains.

In this paper, we evaluate the complete undergraduate coenrollment network over a decade of education at a large American public university. We provide descriptive properties of the network, demonstrating that the coenrollment networks evaluated follow power-law degree distributions similar to many other large-scale networks; that they reveal strong performance-based assortativity; and that network-based features can significantly improve GPA-based student performance predictors. We then implement a network-based, multi-view classification model to predict students' final course grades. In particular, we adapt a structural modeling approach from [19, 34], whereby we model the university-wide undergraduate coenrollment network as an undirected graph. We compare the performance of our predictor to traditional methods used for grade prediction in undergraduate university courses, and demonstrate that a multi-view ensembling approach outperforms both prior "flat" and network-based models for grade prediction across several classification metrics. These findings demonstrate the usefulness of combining diverse approaches in models of student success, and demonstrate specific network-based modeling strategies which are likely to be most effective for grade prediction.

There has been a growing trend in higher education towards increased use and adoption of Massive Open Online Courses (MOOCs). Despite this interest in learning at scale, limited work has compared MOOC activity across subsequent course offerings. In this study, we explore forum activity in ten iterations of the same MOOC. Our results suggest that participation in MOOC forums has changed over the past four years of delivery. First, overall participation in MOOC forums have decreased. Second, in later iterations cohorts of more committed forum users start to resemble formal online courses in size (67>n>36). However, despite the smaller groups of learners that should find it easier to form connections with one another, our analysis did not reveal the expected increase in the quality of social activity. Instead, MOOC forums evolved into smaller on-task question and answer (Q&A) spaces, not capitalizing on the opportunities for social learning. We discuss practical and research implications of such changes.

We present a case study in predictive model replication for student dropout in Massive Open Online Courses (MOOCs) using a large and diverse dataset (133 sessions of 28 unique courses offered by two institutions). This experiment was run on the MOOC Replication Framework (MORF), which makes it feasible to fully replicate complex machine learned models, from raw data to model evaluation. We provide an overview of the MORF platform architecture and functionality, and demonstrate its use through a case study. In this replication of [41], we contextualize and evaluate the results of the previous work using statistical tests and a more effective model evaluation scheme. We find that only some of the original findings replicate across this larger and more diverse sample of MOOCs, with others replicating significantly in the opposite direction. Our analysis also reveals results which are highly relevant to the prediction task which were not reported in the original experiment. This work demonstrates the importance of replication of predictive modeling research in MOOCs using large and diverse datasets, illuminates the challenges of doing so, and describes our freely available, open-source software framework to overcome barriers to replication.

Effective communication is crucial for instructors and students in programming courses. However, communicating about code can be difficult --- particularly in asynchronous settings where an instructor authors an explanation meant to be read and understood by a student later on. Communicating about code is uniquely difficult for two reasons. First, because of the dichotomous nature of the explanation, which consists of fragments of code and natural language descriptions. Second, instructors' explanations of code often involve modifying code throughout their explanation. This paper introduces chat.codes, a new tool for creating guided explanations about code. chat.codes introduces two features that make it easier to communicate about code. First, it adds deictic code references that allows instructors to write messages that reference specific regions of code. Second, it tracks and summarizes code edits in-line with messages, allowing instructors to create explanations in stages. An evaluation showed that these features were beneficial for both instructors and students.

The scholarly investigation of discourse in teaching and learning is multi-disciplinary, theoretically rich, and highly technical. Researchers with backgrounds in education, cognitive psychology, computer science, and the social sciences apply a diverse set of techniques to understand how student discussions affect learning. Aided by big data coming from learning content management and massive open online course systems, these researchers have an unparalleled opportunity for insight into the teaching and learning process. In this paper we summarize some of the challenges and opportunities arising out of three workshops on Educational Discourse. These workshops convened both expert and emerging scholars to discuss the (i) ethical, (ii) technical, and (iii) infrastructure barriers to building a research community focused on computer-mediated educational discourse. Of particular note is that while computational infrastructure exists for storing and manipulating educational discourse, there is a need for a sociotechnical infrastructure upon which community members can come together to engage in joint work.

In this work we investigate the use of deep learning for text analysis to measure elements of student thinking related to issues of privilege, oppression, diversity and social justice. We leverage historical expert annotations as well as a large lexical model to create a more generalizable vocabulary for identifying these characteristics in short student writing. We demonstrate the feasibility of this approach, and identify further areas for research.

In this work, we present (1) a methodology for collecting, evaluating, and utilizing human-annotated data about course syllabi in predictive models of student success, and (2) an empirical analysis of the predictiveness of such features as they relate to others in modeling end-of-course grades in traditional higher education courses. We present a two-stage approach to (1) that addresses several challenges unique to the annotation task, and address (2) using variable importance metrics from a series of exploratory models. We demonstrate that the process of supplementing traditional course data with human-annotated data can potentially improve predictive models with information not contained in university records, and highlight specific features that demonstrate these potential information gains.

There is an emerging trend in higher education for the adoption of massive open online courses (MOOCs). However, despite this interest in learning at scale, there has been limited work investigating how MOOC participants have changed over time. In this study, we explore the temporal changes in MOOC learners' language and discourse characteristics. In particular, we demonstrate that there is a clear trend within a course for language in discussion forums to be of both more on-topic and reflective of deep learning in subsequent offerings of a course. We measure this in two ways, and demonstrate this trend through several repeated analyses of different courses in different domains. While not all courses show an increase beyond statistical significance, the majority do, providing evidence that MOOC learner populations are changing as the educational phenomena matures.

Feature extraction and model selection are two essential processes when building predictive models of student success. In this work we describe and demonstrate a statistical approach to both tasks, comparing five modeling techniques (a lasso penalized logistic regression model, naïve Bayes, random forest, SVM, and classification tree) across three sets of features (week-only, summed, and appended). We conduct this comparison on a dataset compiled from 30 total offerings of five different MOOCs run on the Coursera platform. Through the use of the Friedman test with a corresponding post-hoc Nemenyi test, we present comparative performance results for several classifiers across the three different feature extraction methods, demonstrating a rigorous inferential process intended to guide future analyses of student success systems.

Users frequently experience situations in which their ability to differentiate screen colors is affected by a diversity of situations, such as when bright sunlight causes glare, or when monitors are dimly lit. However, designers currently have no way of choosing colors that will be differentiable by users of various demographic backgrounds and abilities and in the wide range of situations where their designs may be viewed. Our goal is to provide designers with insight into the effect of real-world situational lighting conditions on people's ability to differentiate colors in applications and imagery. We therefore developed an online color differentiation test that includes a survey of situational lighting conditions, verified our test in a lab study, and deployed it in an online environment where we collected data from around 30,000 participants. We then created ColorCheck, an image-processing tool that shows designers the proportion of the population they include (or exclude) by their color choices.

The goal of this tutorial is to share data mining tools and techniques used by computer scientists with educational social scientists. We broadly define educational social scientists as being made up of people with backgrounds in the learning sciences, cognitive psychology, and educational research. The learning analytics community is heavily populated with researchers of these backgrounds, and we believe those that find themselves at the intersection of research, theory, and practice have a particular interest in expanding their knowledge of datadriven tools and techniques.

In this paper we examine the issue of selection bias in quasi-experimental (non-randomly controlled) educational studies. We provide background about common sources of selection bias and the issues involved in evaluating the outcomes of quasi-experimental studies. We describe two methods, matched sampling and propensity score matching, that can be used to overcome this bias. Using these methods, we describe their application through one case study that leverages large educational datasets drawn from higher education institutional data warehouses. The contribution of this work is the recommendation of a methodology and case study that educational researchers can use to understand, measure, and reduce selection bias in real-world educational interventions.

As courses become bigger, move online, and are deployed to the general public at low cost (e.g. through Massive Open Online Courses, MOOCs), new methods of predicting student achievement are needed to support the learning process. This paper presents a novel method for converting educational log data into features suitable for building predictive models of student success. Unlike cognitive modelling or content analysis approaches, these models are built from interactions between learners and resources, an approach that requires no input from instructional or domain experts and can be applied across courses or learning environments.

This work investigates whether enrolling in a Massive Open Online Course (MOOC) with friends or colleagues can improve a learner's performance and social interaction during the course. Our results suggest that signing up for a MOOC with peers correlates positively with the rate of course completion, level of achievement, and discussion forum usage. Further analysis seems to suggest that a learner's interaction with their friends compliments a MOOC by acting as a form of self-blended learning.

Demographics factors have been used successfully as predictors of student success in traditional higher education systems, but their relationship to achievement in MOOC environments has been largely untested. In this work we explore the predictive power of user demographics compared to learner interaction trace data generated by students in two MOOCs. We show that demographic information offers minimal predictive power compared to activity models, even when compared to models created very early on in the course before substantial interaction data has accrued.

This paper describes a method we have developed to convert statistical predictive models into visual narratives which explain student classifications. Building off of the work done within the user experience community, we apply the concept of personas to predictive models. These personas provide familiar and memorable descriptions of the learners identified by data mining activities, and bridge the gap between the data scientist and the learning specialist.

This paper considers the issue of the locus of control in adaptive e-learning environments from the perspective of a new stakeholder; the instructional expert. With an ever increasing ability to gain insight into learners based on their online activities, instructors and instructional designers are poised to add value to the process of adaptation, a process normally reserved for either systems designers or the end user. This work describes the design of an e-learning system which provides automated analytics information to these experts for consideration, and then leverages the insights these experts have made as the basis for content and feature adaptation.

People sometimes miss small parts of meetings and need to quickly catch up without disrupting the rest of the meeting. We developed an Accelerated Instant Replay (AIR) Conferencing system for videoconferencing that enables users to catch up on missed content while the meeting is ongoing. AIR can replay parts of the conference using four different modalities: audio, video, conversation transcript, and shared workspace. We performed two studies to evaluate the system. The first study explored the benefit of AIR catch-up during a live meeting. The results showed that when the full videoconference was reviewed (i.e., all four modalities) at an accelerated rate, users were able to correctly recall a similar amount of information as when listening live. To better understand the benefit of full review, a follow-up study more closely examined the benefits of each of the individual modalities. The results show that users (a) preferred using audio along with any other modality to using audio alone, (b) were most confident and performed best when audio was reviewed with all other modalities, (c) compared to audio-only, had better recall of facts and explanations when reviewing audio together with the shared workspace and transcript modalities, respectively, and (d) performed similarly with audio-only and audio with video review.

Video lecture capture is rapidly being deploying in higher-education institutions as a means of increasing student learning, outreach, and experience. Understanding how learners use these systems and relating this use back to pedagogical and institutional goals is a hard issue that has largely been unexplored. This work describes a novel web-based lecture presentation system which contains fine-grained user tracking features. These features, along with student surveys, have been used to help analyse the behaviour of hundreds of students over an academic term, quantifying both the learning approaches of students and their perceptions on learning with lecture capture.

This paper gives a short overview of the Opencast Matterhorn system. Built by an open community of individuals and institutions, Matterhorn provides a lecture capture platform for both research and production environments. Matterhorn is comprehensive and scalable, and includes components for the acquisition, processing, and playback of content. Matterhorn is licensed under the liberal Educational Community License (ECL 2.0), a flexible OSI approved open source license, and the Opencast community is free for all institutions, corporations, or individuals to join.

Guidelines for designing information charts (such as bar charts) often state that the presentation should reduce or remove 'chart junk' - visual embellishments that are not essential to understanding the data. In contrast, some popular chart designers wrap the presented data in detailed and elaborate imagery, raising the questions of whether this imagery is really as detrimental to understanding as has been proposed, and whether the visual embellishment may have other benefits. To investigate these issues, we conducted an experiment that compared embellished charts with plain ones, and measured both interpretation accuracy and long-term recall. We found that people's accuracy in describing the embellished charts was no worse than for plain charts, and that their recall after a two-to-three-week gap was significantly better. Although we are cautious about recommending that all charts be produced in this style, our results question some of the premises of the minimalist approach to chart design.

When people attend meetings they may miss parts of the discussion if they, for example, step out to take a phone call, go to the bathroom, or have a momentary lapse in concentration. As a result, they may need to catch up on what they missed upon returning to the meeting. Asking other attendees for a recap is often disruptive. To avoid such disruptions, we have developed an Accelerated Instant Replay (AIR) Conferencing system for videoconferencing that enables participants to privately catch up to an ongoing meeting. We explored several mechanisms where the meeting content is replayed at an accelerated rate so that the participants can catch up to the live discussion reasonably quickly.

This paper describes work we are doing to identify significant events in video captures of academic lectures. Unlike other approaches which tend to define per-image comparison threshold values based on intuition or empirically derived results, we use supervised machine learning techniques to automatically determine appropriate image characteristics based on end-users understanding of what constitutes an important event. This makes our approach more adaptable to different kinds of content, and still provides a substantial level of agreement with human experts.

This paper describes a low-cost distributed computing approach for the post-processing of videos. It has been implemented to support the creation of composite videos from various video sources, and supports image manipulation on individual frames of the video. The principle deployment scenario for this work has been to support the automatic creation of output video based on traditional face-to-face lectures. Experimental results suggest that this approach can lead to significant speedups with a reduction in operating costs by leveraging existing underutilized equipment.

In our previous work, we developed the LOCO ontology framework which formalizes the notion of learning object context as a complex interplay of learning activities, learning objects, and learners. We now use that framework in conjunction with semantic annotation to generate different kinds of feedback (which we had identified by interviewing several Web educators) for educators to help them improve the learning process in Web-based settings. To test the feasibility of the proposed approach for feedback provision we developed a tool named LOCO-Analyst. Here we report on our experiences in developing LOCO-Analyst and using it to generate feedback out of the real data obtained from the iHelp Courses Learning Content Management System. Finally, we present evaluation results.

Web-based learning involves provision of personal on-line spaces to learners and teachers. Yet due to a lack of proper privacy policies and technical frameworks to implement them, seemingly innocent data transactions can carry risks to privacy in e-learning environments. We investigate privacy issues in e-learning and make recommendations for building environments that enhance privacy but allow features like content personalization and peer- collaboration. iHelp, a learning environment that supports both learners and instructors, implements many of these recommendations.

Learning objects are reusable pieces of educational material intended to be strung together to form larger educational units such as activities, lessons, or whole courses. These materials are stored in learning object repositories which can be distributed in nature. We outline the issues associated with creating derivative works based on learning objects in a general manner, and discuss the support that exists within current metadata specifications.

An important issue in reusing learning objects on the semantic Web is the development of appropriate technology to facilitate the discovery and reuse of learning objects stored in global and local repositories. Another issue is the development of ontologies for marking up the structure of learning objects and ascribing pedagogical meaning to them so that they can be understandable by machines. A third issue is making learning objects smarter so that they can perform a more meaningful role on the semantic Web. We discuss these and other issues as they affect the exploitation of learning objects on the semantic Web.