Building teams that foster equitable interaction provides the foundation for a positive collaborative learning experience.Existing literature shows that many context-specific algorithms exist to help instructors form teams automatically in large classes, but the field lacks general guidelines for selecting a suitable algorithm in a given pedagogical context and lacks a general evaluation approach that allows for the methodological comparison of these algorithms.This paper presents a general-purpose team formation algorithm that considers diversity and inclusion in its design.We also describe an evaluation framework with diversity metrics to assess team compositions using synthetically generated student data and real class data.Our simulation and classroom experiments show that our algorithm performs competitively against three state-of-theart algorithms.We hope this work contributes to building a more equitable and collaborative learning environment for students. CCS Concepts• Human-centered computing → Collaborative and social computing design and evaluation methods; • General and reference → Metrics; • Social and professional topics → User characteristics.

This research-to-practice full paper presents guiding principles for developing assessments based on our experience in teaching the software engineering capstone course over the last twelve years. Software engineering courses are central to computer science and engineering programs. To provide students with an authentic learning experience, teams of students work on realistic projects that help them apply theoretical concepts to develop practical skills. Challenges arise with increasing class sizes and limited teaching resources. Despite these constraints, educators must intentionally design assessments that align with learning theories that promote deeper learning opportunities and support lifelong learning. In this work, we report on our experience designing and adapting the assessments used in our software engineering capstone course over the past twelve years. We reflect on our approach by aligning the evaluation strategies to learning theories such as behaviorism, constructivism, and social constructivism. This research-to-practice paper discusses the practical implications behind different assessment strategies in the face of large class sizes and presents guiding principles for developing assessments for software engineering team projects.

Ahstract–This innovative practice full paper describes a case study from a software engineering capstone project course. Undergraduate programs often have a final-year capstone course designed to integrate and apply the knowledge and skills students have previously acquired while adapting to industry-standard practices. Capstones play a critical role in bridging the gap between academia and the industry as students transition to the workforce. Over the past twelve years, our institution has adopted a client-based model where industry clients work closely with a single team to solve a real-world problem. However, rising enrollment has put a strain on running this model effectively because of difficulties in recruiting clients, managing numerous client relationships simultaneously, and keeping client-student interactions sustainable. To tackle these challenges, we propose a new client model where clients pitch their ideas as themes in a competition and act as panel judges in evaluating student team submissions. We call this the hackathon client model and evaluate it in a class with 22 teams and 104 students. Through a thematic analysis of the qualitative responses gathered from this study, our findings suggest this new model provides a scalable alternative to operating a large capstone class while preserving many of the benefits of the traditional client model. However, both students and clients indicate having more means of communication would improve the project requirements phase and strengthen their relationship. We discuss ideas on improving the hackathon client model and plans for future experimentation in large capstones.

This research-to-practice full paper investigates using team process analytics from GitHub to support team management. Effective teamwork is essential in higher education learning and workplace success. The role of educators in supporting proper team functioning includes helping students learn how to participate actively and communicate effectively in meetings, delegate work fairly, manage high-quality work throughput, and resolve conflicts if problems arise. Problems often emerge when team members have differing visions or individuals do not contribute equally to the work output. These problems are exacerbated in large classes involving many teams. Detecting these potential issues in teams and helping students work through them is important for team success. In software engineering projects, monitoring individual contributions can begin with mining activities on programming platforms such as GitHub, which makes much of the individual contributions more visible and quantifiable. In this work, we propose a framework for fairly assessing teamwork and present the development of team analytics to assist educators in detecting potential issues in team collaboration. We describe a pilot study involving this tool in the context of a software engineering capstone course with 104 students split into 22 teams managed by 4 teaching assistants. Our results show that the reports offer value in guiding the evaluation process and identifying where problems may be, but do not replace the actual repository analysis where needed. We discuss the potential value of using this tool to improve collaboration.

Understanding team effectiveness is crucial to improving performance in many workplace and educational contexts. Many theories have postulated a variety of factors that influence team success. However, they are limited to a descriptive framework or involve small empirical studies. In contexts involving many teams, we would ideally like to monitor ongoing team behaviors to alert problematic behaviors and reward positive actions. To accomplish this goal, we propose to develop a computational model of team concepts to facilitate the detection, prediction, and proper management of team behaviors. In this work, we synthesize the literature on team models and present six overarching team concepts. We select two specific concepts and model them as a dynamic Bayesian network (DBN). We demonstrate the utility of the DBN models in simulation and discuss the gap between the behaviors prescribed by team theories and the data needs in computational models. Lastly, we discuss possible data sources that serve as starting points for developing empirically accurate models.

Teaching design thinking and human-computer interaction (HCI) from technical disciplines are challenging due to the need for frequent content refreshes using up-to-date technology examples and students' negative preconceptions about the material. In particular, students typically find the course content too easy and the grading too subjective. Beyond these issues, the problems in our HCI course are exacerbated by the diversity of academic backgrounds in the student population. Over the years, we have struggled to deliver the course material at a pace that is appropriate for everyone and to develop assessments that can fairly evaluate the relevant design and technical skills involved. Here, we propose a personalized learning approach to tackle the problem of delivering content that suits students with diverse academic backgrounds. Specifically, we designed four features to personalize assessments in the HCI course: alternative pathways, flexible timing, multiple test attempts, and choice in programming options. Our study with 360 students showed a high uptake in all the personalization features and a positive impact on student performance and perception of the course. This case study describes how we implemented these personalized learning features successfully in a large university class to improve the student's learning experience. We discuss implementation tradeoffs and the generalizability of these features.

JUnit++ is an extension to the JUnit 5 test framework designed specifically for testing student submissions in CS1 classes. JUnit++ simplifies the test writing process by eliminating repetitive code and letting test writers focus on the logical correctness and structure of the solution. Although everything our extension does can be achieved by using only the base JUnit framework, we have found that using our extension speeds up the efficiency of the test development process and makes it easier to review and maintain tests.

Many educational resources are available that teach children computational thinking and visual programming. As part of this initiative, we develop computational puzzles from well-known computational thinking activities. Inspired by the idea of geocaching, we made our puzzles accessible and fun for the whole family by embedding them into a scavenger hunt. We describe our approach that frames computational puzzles as an outdoor family activity. Our project was launched in 2021 and has received informal positive feedback.

Students often find CS1 to be difficult and the workload too demanding. Those who are struggling may lose marks on assessments due to knowledge gaps, slips, or simply a lack of time. Meanwhile, other students demand more practice questions to better prepare for upcoming exams. To mitigate these differences, we developed an open learning platform that combines ideas from mastery learning, concept mapping, and gamification. The main feature of this work is an open question bank with over 1,500 multiple-choice questions, Parsons problems, and programming exercises for CS1. Using our platform, students can earn tokens by solving questions in multiple ways, including free-form practice, setting personal goals, and completing challenges individually or in teams. Our platform also has a leaderboard to promote engagement and personal analytics to foster self-regulated learning and deliberate practice. Initial feedback from over 500 students has been encouraging. Ultimately, we hope that our system can accommodate different student preferences in how they want to learn and practice programming.

Mastery learning and deliberate practice promote personalized learning, allowing the learner to improve through a repetitive and targeted approach. Unfortunately, this pedagogy is challenging to implement in classrooms where everyone is expected to learn at the same pace. Various studies have successfully demonstrated that certain aspects of mastery learning can be integrated into the curriculum. We adopt a similar pedagogical strategy and explain our implementation approach with online assessments. Since this is a new pedagogical approach to assessing student learning, we collected data to investigate test-taking behavior and evaluated potential learning gains in this new test format. As part of this endeavor, we developed a model to detect trial-and-error sequences in test attempts. Our results point to a small percentage of guessing behavior, which is encouraging evidence supporting this test approach is a viable way to implement mastery learning in our curriculum.

Skills analysis is an interdisciplinary area that studies labor market trends and provides recommendations for developing educational standards and re-skilling efforts. We leverage techniques in this area to develop a scalable approach that identifies and evaluates educational competencies. In this work, we developed a skills extraction algorithm that uses natural language processing and machine learning techniques. We evaluated our algorithm on a labeled dataset and found its performance to be competitive with state-of-the-art methods. Using this algorithm, we analyzed student skills, university course syllabi, and online job postings. Our cross-sector analysis provides an initial landscape of skill needs for specific job titles. Additionally, we conducted a within-sector analysis based on programming jobs, computer science curriculum, and undergraduate students. Our findings suggest that students have a variety of hard skills and soft skills, but they are not necessarily the ones that employers want. The data also suggests these courses teach skills that are somewhat different from industry needs, and there is a lack of emphasis on soft skills. These results provide an initial assessment of the program competencies for a computer science program. Future work includes more data gathering, improving the algorithm, and applying our method to assess additional educational programs.

To keep young students engaged in computer science, it is crucial to develop teaching material that they find interesting and relevant. Unfortunately, standard CS1 textbooks typically use examples that are uninspiring or inaccessible to young people. To better understand the disparity between textbook examples and student interests, we analyzed a collection of CS1 textbooks and compared the resulting topics to those elicited from young students via focus groups. We found 47% of textbook topics (out of 53 topics from 910 code examples) did not overlap with any topic mentioned by our participants. Conversely, among the topics elicited from the participants, we found 29% of these topics (out of 24 topics from 1936 items) missing from textbooks. To measure the overlap between these two data samples, we computed the Bhattacharyya coefficient and obtained 0.4452 indicating a strong difference between the two sets. These results lead us to advocate for changes in the teaching materials in order to make them more engaging for young students.

Emphasis on 21stcentury skills has placed much importance on providing students with collaboration opportunities, despite the resistance by some students who prefer to work alone. In order to facilitate a flexible learning environment that fosters both individual productivity as well as collaborative problem solving, we designed a study to better understand the factors influencing students' choice to collaborate in an online setting. We developed a web-based learning software for practicing linked list exercises and conducted a study with 67 participants in a CS2 class. Our results indicate that online collaboration provides a peer learning opportunity for students with lower confidence to become more comfortable with the material. Moreover, we analyzed student data and report on the performance tradeoffs (speed vs. number of mistakes) between working collaboratively and working solo.

Adaptive software systems are intended to modify their appearance, performance or functionality to the needs and preferences of different users. A key bottleneck in building effective adaptive systems is accounting for the cost of disruption to a user's mental model of the application caused by the system's adaptive behaviour. In this work, we propose a probabilistic approach to modeling the cost of disruption. This allows an adaptive system to tradeoff disruption cost with expected savings (or other benefits) induced by a potential adaptation in a principled, decision-theoretic fashion. We conducted two experiments with 48 participants to learn model parameters in an adaptive menu selection environment. We demonstrate the utility of our approach in simulation and usability studies. Usability results with 8 participants suggest that our approach is competitive with other adaptive menus w.r.t. task performance, while providing the ability to reduce disruption and adapt to user preferences.

The development of intelligent assistants has largely benefited from the adoption of decision-theoretic (DT) approaches that enable an agent to reason and account for the uncertain nature of user behaviour in a complex software domain. At the same time, most intelligent assistants fail to consider the numerous factors relevant from a human-computer interaction perspective. While DT approaches offer a sound foundation for designing intelligent agents, these systems need to be equipped with an interaction cost model in order to reason the impact of how (static or adaptive) interaction is perceived by different users. In a DT framework, we formalize four common interaction factors --- information processing, savings, visual occlusion, and bloat. We empirically derive models for bloat and occlusion based on the results of two users experiments. These factors are incorporated in a simulated help assistant where decisions are modeled as a Markov decision process. Our simulation results reveal that our model can easily adapt to a wide range of user types with varying preferences.

Automated software customization is drawing increasing attention as a means to help users deal with the scope, complexity, potential intrusiveness, and ever-changing nature of modern software. The ability to automatically customize functionality, interfaces, and advice to specific users is made more difficult by the uncertainty about the needs of specific individuals and their preferences for interaction. Following recent probabilistic techniques in user modeling, we model our user with a dynamic Bayesian network (DBN) and propose to explicitly infer the type --- a composite of personality and affect variables --- in real time. We design the system to reason about the impact of its actions given the user's current attitudes. To illustrate the benefits of this approach, we describe a DBN model for a text-editing help task. We show, through simulations, that user types can be inferred quickly, and that a myopic policy offers considerable benefit by adapting to both different types and changing attitudes. We then develop a more realistic user model, using behavioural data from 45 users to learn model parameters and the topology of our proposed user types. With the new model, we conduct a usability experiment with 4 users and 4 different policies. These experiments, while preliminary, show encouraging results for our adaptive policy.

Software customization has been argued to benefit both the productivity of software engineers and end users. However, most customization methods rely on specialists to manually tweak individual applications for a specific user group. Existing software development methods also fail to acknowledge the importance of different kinds of user skills and preferences and how these might be incorporated into a customizable software design. We propose a framework for performing requirements analysis on user goals, skills, and preferences in order to generate a customizable software design. We illustrate our methodology with an email system and review an on-going case study involving users with traumatic brain injury.