In undergraduate computing degree programs, students typically participate in team-based capstone projects to develop real-world software products. While these projects can provide students with authentic learning experiences involving various aspects of software development, project management, and teamwork, individual contributions are rarely assessed. Instead, instructors typically evaluate all members of a team together using a set of traditional deliverables as the basis for grading. This approach is deficient in that it deprives students of opportunities to focus on specific aspects of software development, to obtain and provide feedback, and to reflect on individual learning experiences. In this Position & Curricula Initiative (PCI) paper, we propose a pedagogy to facilitate students' ability to choose, document and reflect on individual project contributions, receive feedback specific to those contributions, and assess peer contributions. A novel framework of performance indicators—based around ABET's student learning outcomes for undergraduate computing and engineering degree programs—is presented that describes measurable achievements and provides evidence of attaining student learning outcomes. Portfolio creation and assessment activities are outlined, including the use of a software tool to help streamline the process for instructors and students alike. This structured pedagogical framework allows students the ability to engage more directly in the set of software development and project management skills they are most interested in, thus increasing their motivation to succeed and ultimately their preparation for careers in the software profession.

Metacognition is widely acknowledged as a key soft skill in collaborative software development. The ability to plan, monitor, and reflect on cognitive and team processes is crucial to the efficient and effective functioning of a software team. To explore students' use of reflection--one aspect of metacognition--in undergraduate team software projects, we analyzed the online chat channels of teams participating in agile software development projects in two undergraduate courses that took place exclusively online (n = 23 teams, 117 students, and 4,915 chat messages). Teams' online chats were dominated by discussions of work completed and to be done; just two percent of all chat messages showed evidence of reflection. A follow-up analysis of chat vignettes centered around reflection messages (n = 63) indicates that three-fourths of the those messages were prompted by a course requirement; just 14% arose organically within the context of teams' ongoing project work. Based on our findings, we identify opportunities for computing educators to increase, through pedagogical and technological interventions, teams' use of reflection in team software projects.

Assessing team software development projects is notoriously difficult and typically based on subjective metrics. To help make assessments more rigorous, we conducted an empirical study to explore relationships between subjective metrics based on peer and instructor assessments, and objective metrics based on GitHub and chat data. We studied 23 undergraduate software teams ( n = 117 students) from two undergraduate computing courses at two North American research universities. We collected data on teams’ (a) commits and issues from their GitHub code repositories, (b) chat messages from their Slack and Microsoft Teams channels, (c) peer evaluation ratings from the CATME peer evaluation system, and (d) individual assignment grades from the courses. We derived metrics from (a) and (b) to measure both individual team members’ contributions to the team, and the equality of team members’ contributions. We then performed Pearson analyses to identify correlations among the metrics, peer evaluation ratings, and individual grades. We found significant positive correlations between team members’ GitHub contributions, chat contributions, and peer evaluation ratings. In addition, the equality of teams’ GitHub contributions was positively correlated with teams’ average peer evaluation ratings and negatively correlated with the variance in those ratings. However, no such positive correlations were detected between the equality of teams’ chat contributions and their peer evaluation ratings. Our study extends previous research results by providing evidence that (a) team members’ chat contributions, like their GitHub contributions, are positively correlated with their peer evaluation ratings; (b) team members’ chat contributions are positively correlated with their GitHub contributions; and (c) the equality of team’ GitHub contributions is positively correlated with their peer evaluation ratings. These results lend further support to the idea that combining objective and subjective metrics can make the assessment of team software projects more comprehensive and rigorous.

Providing students with authentic software development experiences is essential to preparing them for careers in industry. To that end, many undergraduate courses include a team-based software development experience in which each team works on a different software project. This raises significant challenges for assessing student work and measuring the impact of pedagogical interventions: What do we measure and how, when each team is working on a different project? To address this question, we present a collection of metrics developed using the Goal-Question-Metric framework from the empirical software engineering literature, and an empirical study in which we applied those metrics to assess 23 team software projects involving 94 students at three institutions. Study results suggest that these metrics, which gauge commit, issue, and overall product quality, are sensitive to differences in the quality of teams' processes and products. This work contributes a new metric-based approach to evaluating key aspects of software development processes and products in a wide variety of computing courses.

Student retention and learning in STEM disciplines is a growing problem. The 2012 report by the US President's Council of Advisors on Science and Technology (PCAST) predicts a future deficit in science, engineering, and mathematics (STEM) in the following decade and emphasizes the importance of addressing this issue. With this as a motivating factor, the OSBLE+ Social Programming Environment (SPE) was used to leverage social and programming data for the basis of automatically generated prompts inserted into the SPE. These prompts were designed to stimulate help-seeking, help-giving, and social interaction in the learning environment. A social network analysis was performed in order to determine whether exposure to the automated interventions would positively affect the relationship among students over time. Results of this study suggest that students in the experimental treatment who were presented with automated prompts developed more connected and social networks than those in the control treatment.

It has been well documented that a large portion of the cost of any software lies in the time spent by developers in understanding a program's source code before any changes can be undertaken. One of the main contributors to software comprehension, by subsequent developers or by the authors themselves, has to do with the quality of the lexicon, (i.e., the identifiers and comments) that is used by developers to embed domain concepts and to communicate with their teammates. In fact, previous research shows that there is a positive correlation between the quality of identifiers and the quality of a software project. Results suggest that poor quality lexicon impairs program comprehension and consequently increases the effort that developers must spend to maintain the software. However, we do not yet know or have any empirical evidence, of the relationship between the quality of the lexicon and the cognitive load that developers experience when trying to understand a piece of software. Given the associated costs, there is a critical need to empirically characterize the impact of the quality of the lexicon on developers' ability to comprehend a program.

Analyzing the process data of students as they complete programming assignments has the potential to provide computing educators with insights into both their students and the processes by which they learn to program. In prior research, we explored the relationship between (a) students’ programming behaviors and course outcomes, and (b) students’ participation within an online social learning environment and course outcomes. In both studies, we developed statistical measures derived from our data that significantly correlate with students’ course grades. Encouraged both by social theories of learning and a desire to improve the accuracy of our statistical models, we explore here the impact of incorporating our predictive measure derived from social behavior into three separate predictive measures derived from programming behaviors. We find that, in combining the measures, we are able to improve the overall predictive power of each measure. This finding affirms the importance of social interaction in the learning process, and provides evidence that predictive models derived from multiple sources of learning process data can provide significantly better predictive power by accounting for multiple factors responsible for student success.

Educators stand to benefit from advance predictions of their students' course performance based on learning process data collected in their courses. Indeed, such predictions can help educators not only to identify at-risk students, but also to better tailor their instructional methods. In computing education, at least two different measures, the Error Quotient and Watwin Score, have achieved modest success at predicting student course performance based solely on students' compilation attempts. We hypothesize that one can achieve even greater predictive power by considering students' programming activities more holistically. To that end, we derive the Normalized Programming State Model (NPSM), which characterizes students' programming activity in terms of the dynamically-changing syntactic and semantic correctness of their programs. In an empirical study, the NPSM accounted for 41% of the variance in students' programming assignment grades, and 36% of the variance in students' final course grades. We identify the components of the NPSM that contribute to its explanatory power, and derive a formula capable of predicting students' course programming performance with between 36 and 67 percent accuracy, depending on the quantity of programming process data.

Social learning theory emphasizes the importance of providing learners with opportunities to observe their peers, and to participate actively in a community. Unfortunately, early computing courses tend to emphasize individual programming assignments, which discourage learners from observing and working with their peers. In order to explore the possibility that increased opportunities for social awareness and interaction while working on programming assignments might influence learning outcomes in early computing courses, we are studying the design and use of social networking-style activity streams in such courses. In an empirical study of the use of two types of activity streams in a CS 2 course - one that was part of a learning management system, and one integrated directly into students' programming environment - we found that students who used the integrated stream were twice as socially active; however, social participation in both environments was positively correlated with students' grades. Our results suggest that the use of activity streams as an adjunct to individual programming assignments can positively influence learning; computing instructors would do well to find ways to get their students to participate actively in activity streams during the programming process.

A meta-analysis on the effectiveness of Intelligent Tutoring Systems (ITS) in computer science education compared the learning outcomes of ITS and non-ITS instruction. A search of the literature found 22 effect sizes (involving 1,447 participants) that met the pre-defined inclusion criteria. Although most of the ITS were used to teach programming, other topics such as database design and computer literacy were also represented. There was a significant overall effect size favoring the use of ITS. There was a significant advantage of ITS over teacher-led classroom instruction and non-ITS computer-based instruction. ITS were more effective than the instructional methods to which they were compared regardless of whether they modeled misconceptions and regardless of whether they were the primary means of instruction or were an integrated component of learning activities that included other means of instruction.

Changes in students' personal epistemology are especially important for engineering educators to examine because they may affect the way students learn and their ability to adapt to engineering education learning environments and culture. Despite the large amount of theoretical research that has been done concerning students' conceptual change, little research has been done concerning their epistemological change. This study is a preliminary effort in identifying students' beliefs about knowledge and their effect on learning and successful comprehension of engineering concepts. The cohort consisted of 10 Civil Engineering students in a sophomore level Statics class, and each student participated in a 90-minute interview. The questions were based around an epistemological framework currently in development, which includes six separate dimensions but allows for examination of the ties between the dimensions. Analysis was completed in multiple stages, and involved two researchers co-coding for inter-relater reliability. Most students viewed knowledge as very simple, certain and objective. However, many students felt that different people had different knowledge of statics because of their different backgrounds and learning. These beliefs may be leveraged to support pedagogical practices of proven effectiveness such as peer-tutoring or other active learning methods.