In undergraduate computing degree programs, students typically participate in team-based capstone projects to develop real-world software products. While these projects can provide students with authentic learning experiences involving various aspects of software development, project management, and teamwork, individual contributions are rarely assessed. Instead, instructors typically evaluate all members of a team together using a set of traditional deliverables as the basis for grading. This approach is deficient in that it deprives students of opportunities to focus on specific aspects of software development, to obtain and provide feedback, and to reflect on individual learning experiences. In this Position & Curricula Initiative (PCI) paper, we propose a pedagogy to facilitate students' ability to choose, document and reflect on individual project contributions, receive feedback specific to those contributions, and assess peer contributions. A novel framework of performance indicators—based around ABET's student learning outcomes for undergraduate computing and engineering degree programs—is presented that describes measurable achievements and provides evidence of attaining student learning outcomes. Portfolio creation and assessment activities are outlined, including the use of a software tool to help streamline the process for instructors and students alike. This structured pedagogical framework allows students the ability to engage more directly in the set of software development and project management skills they are most interested in, thus increasing their motivation to succeed and ultimately their preparation for careers in the software profession.

When graduates of computing degree programs enter the software industry, they will most likely join teams working on legacy code bases developed by people other than themselves. In these so-called brownfield software development settings, generative artificial intelligence (GenAI) coding assistants like GitHub Copilot are rapidly transforming software development practices, yet the impact of GenAI on student programmers performing brownfield development tasks remains underexplored. This paper investigates how GitHub Copilot influences undergraduate students' programming performance, behaviors, and understanding when completing brownfield programming tasks in which they add new code to an unfamiliar code base. We conducted a controlled experiment in which 10 undergraduate computer science students completed highly similar brownfield development tasks with and without Copilot in a legacy web application. Using a mixed-methods approach combining performance analysis, behavioral analysis, and exit interviews, we found that students completed tasks 35% faster (p < 0.05) and made 50% more solution progress p (< 0.05) when using Copilot. Moreover, our analysis revealed that, when using Copilot, students spent 11% less time manually writing code (p < 0.05), and 12% less time conducting web searches (p < 0.05), providing evidence of a fundamental shift in how they engaged in programming. In exit interviews, students reported concerns about not understanding how or why Copilot suggestions work. This research suggests the need for computing educators to develop new pedagogical approaches that leverage GenAI assistants' benefits while fostering reflection on how and why GenAI suggestions address brownfield programming tasks. Complete study results and analysis are presented at https://ghcopilot-icer.github.io/.

This work examines the state-of-the-art of cybersecurity education in the United States by considering two sources of data. The first source consists of Programs of Study for cybersecurity programs at Centers of Academic Excellence in Cybersecurity designated by the National Security Agency. Statistics were aggregated from a sample of one hundred CAE-C institutions, trends and gaps are identified, and improvements are proposed. The second source is peer-reviewed research published in the field of cybersecurity education over the last decade. A review of this literature shows a strong focus on identifying instructional content and developing educational tools while simultaneously indicating a shortage of research into rigorous evaluation of the instructional approaches being used to teach cybersecurity. Our review of these two sources of data highlight two paths to improving cybersecurity education in the United States. First, institutions offering cybersecurity degrees could work more closely with groups such as NIST, ACM, and IEEE to ensure their curricula match the needs of industry and they are graduating work-ready cybersecurity specialists. While CAE-C designation provides certain requirements for the amount of cybersecurity content included in curricula, designated institutions vary widely in the types of programs they offer and how many cybersecurity-specific courses they provide. Second, cybersecurity education could benefit from an influx of ideas from educational psychology regarding instructional theories such as cognitive load theory.

Metacognition is widely acknowledged as a key soft skill in collaborative software development. The ability to plan, monitor, and reflect on cognitive and team processes is crucial to the efficient and effective functioning of a software team. To explore students' use of reflection--one aspect of metacognition--in undergraduate team software projects, we analyzed the online chat channels of teams participating in agile software development projects in two undergraduate courses that took place exclusively online (n = 23 teams, 117 students, and 4,915 chat messages). Teams' online chats were dominated by discussions of work completed and to be done; just two percent of all chat messages showed evidence of reflection. A follow-up analysis of chat vignettes centered around reflection messages (n = 63) indicates that three-fourths of the those messages were prompted by a course requirement; just 14% arose organically within the context of teams' ongoing project work. Based on our findings, we identify opportunities for computing educators to increase, through pedagogical and technological interventions, teams' use of reflection in team software projects.

Assessing team software development projects is notoriously difficult and typically based on subjective metrics. To help make assessments more rigorous, we conducted an empirical study to explore relationships between subjective metrics based on peer and instructor assessments, and objective metrics based on GitHub and chat data. We studied 23 undergraduate software teams ( n = 117 students) from two undergraduate computing courses at two North American research universities. We collected data on teams’ (a) commits and issues from their GitHub code repositories, (b) chat messages from their Slack and Microsoft Teams channels, (c) peer evaluation ratings from the CATME peer evaluation system, and (d) individual assignment grades from the courses. We derived metrics from (a) and (b) to measure both individual team members’ contributions to the team, and the equality of team members’ contributions. We then performed Pearson analyses to identify correlations among the metrics, peer evaluation ratings, and individual grades. We found significant positive correlations between team members’ GitHub contributions, chat contributions, and peer evaluation ratings. In addition, the equality of teams’ GitHub contributions was positively correlated with teams’ average peer evaluation ratings and negatively correlated with the variance in those ratings. However, no such positive correlations were detected between the equality of teams’ chat contributions and their peer evaluation ratings. Our study extends previous research results by providing evidence that (a) team members’ chat contributions, like their GitHub contributions, are positively correlated with their peer evaluation ratings; (b) team members’ chat contributions are positively correlated with their GitHub contributions; and (c) the equality of team’ GitHub contributions is positively correlated with their peer evaluation ratings. These results lend further support to the idea that combining objective and subjective metrics can make the assessment of team software projects more comprehensive and rigorous.

As in other STEM disciplines, early computing courses tend to stress individual assignments and discourage collaboration. This can lead to negative learning experiences that compel some students to give up. According to social learning theory, one way to improve students’ learning experiences is to help them form and participate actively in vibrant social learning communities. Building on social learning theory, we have designed a set of software interventions (scaffolds and prompts) that leverage automatically collected learning process data to promote increased social interactions and better learning outcomes in individual programming assignments, which are a key component of early undergraduate computing courses. In an empirical study, we found that students’ interaction with the interventions was correlated with increased social activity, improved attitudes toward peer learning, more closely coupled social networks, and higher performance on programming assignments. Our work contributes a theoretically motivated technological design for social programming interventions; an understanding of computing students’ willingness to interact with the interventions; and insights into how students’ interactions with the interventions are associated with their social behaviors, attitudes, connectedness with others in the class, and their course outcomes.

Effective self-regulated learning (SRL) is important to student academic success. Understanding what SRL struggles students face in programming assignments is critical to guide many efforts in computing education, such as designing scalable interventions and developing effective learning technologies. Prior studies on this topic contributed to understanding what SRL strategies CS students typically use in programming assignments, and the interventions for some SRL struggles such as procrastination. However, few studies have investigated student SRL struggles in programming systematically. To fill this gap, we investigate student SRL struggles in the context of CS2 through a case study. We used multiple approaches to collect real-time data and validate our findings, such as tracking student progress, identifying potential SRL struggles, and interviewing identified struggling students to confirm our identifications. This study contributes to a deeper understanding of what SRL struggles students face in programming at a fine-grained level, and provides guidance on interventions for SRL struggles.

Providing students with authentic software development experiences is essential to preparing them for careers in industry. To that end, many undergraduate courses include a team-based software development experience in which each team works on a different software project. This raises significant challenges for assessing student work and measuring the impact of pedagogical interventions: What do we measure and how, when each team is working on a different project? To address this question, we present a collection of metrics developed using the Goal-Question-Metric framework from the empirical software engineering literature, and an empirical study in which we applied those metrics to assess 23 team software projects involving 94 students at three institutions. Study results suggest that these metrics, which gauge commit, issue, and overall product quality, are sensitive to differences in the quality of teams' processes and products. This work contributes a new metric-based approach to evaluating key aspects of software development processes and products in a wide variety of computing courses.

Interactive textbooks have great potential to increase student engagement with the course content which is critical to effective learning in computing education. Prior research on digital textbooks and interactive visualizations contributes to our understanding of student interactions with visualizations and modeling textbook knowledge concepts. However, research investigating student usage of interactive computing textbooks is still lacking. This study seeks to fill this gap by modeling student engagement with a Jupyter-notebook-based interactive textbook. Our findings suggest that students' active interactions with the presented interactive textbook, including changing, adding, and executing code in addition to manipulating visualizations, are significantly stronger in predicting student performance than conventional reading metrics. Our findings contribute to a deeper understanding of student interactions with interactive textbooks and provide guidance on the effective usage of said textbooks in computing education.

The 32 chapters of the 2019 Cambridge Handbook of Computing Education Research synthesize the existing research in computing education and propose new directions for future research. An author from each chapter will summarize their chapter with auto-advancing slides. Attendees will be introduced to the breadth of content in the new handbook and can identify chapters of interest. This fits uniquely as a special session, and will likely be informative, inspiring, and overwhelming.

Student retention and learning in STEM disciplines is a growing problem. The 2012 report by the US President's Council of Advisors on Science and Technology (PCAST) predicts a future deficit in science, engineering, and mathematics (STEM) in the following decade and emphasizes the importance of addressing this issue. With this as a motivating factor, the OSBLE+ Social Programming Environment (SPE) was used to leverage social and programming data for the basis of automatically generated prompts inserted into the SPE. These prompts were designed to stimulate help-seeking, help-giving, and social interaction in the learning environment. A social network analysis was performed in order to determine whether exposure to the automated interventions would positively affect the relationship among students over time. Results of this study suggest that students in the experimental treatment who were presented with automated prompts developed more connected and social networks than those in the control treatment.

In order to meet the health needs of the coming "age wave," technology needs to be designed that supports remote health monitoring and assessment. In this study we design clinician in the loop (CIL), a clinician-in-the-loop visual interface, that provides clinicians with patient behavior patterns, derived from smart home data. A total of 60 experienced nurses participated in an iterative design of an interactive graphical interface for remote behavior monitoring. Results of the study indicate that usability of the system improves over multiple iterations of participatory design. In addition, the resulting interface is useful for identifying behavior patterns that are indicative of chronic health conditions and unexpected health events. This technology offers the potential to support self-management and chronic conditions, even for individuals living in remote locations.

As is the case for many undergraduate STEM degree programs, computing degree programs are plagued by high attrition rates. This is especially true in early computing courses, in which failure and drop-out rates in the 35 to 50 percent range are common. By collecting learning process data as students engage in computer programming assignments, computing educators can place themselves in a position not only to better understand students' struggles, but also to better tailor instructional interventions to students' needs. We have developed OSBLE+, a learning management and analytics environment that interfaces with a computer programming environment to support the automatic collection of learners' programming process and social data as they work on programming assignments, while also providing an interactive environment for the analysis and visualization of those data. In ongoing work, we are using OSBLE+ to explore two possibilities: (a) leveraging learning and social data to strategically deliver automated learning interventions, and (b) presenting learners with visual representations of their learning data in order to prompt them to reflect on and discuss their learning processes.

Analyzing the process data of students as they complete programming assignments has the potential to provide computing educators with insights into their students and the processes by which they learn to program. In prior work, we developed a statistical model that accurately predicts students' homework grades. In this paper, we investigate the relationship between the paths that students take through the programming states on which our statistical model is based, and their overall course achievement. Examining the frequency of the most common transition paths revealed significant differences between students who earned A's, B's, and C's in a CS 2 course. Our results indicate that a) students of differing achievement levels approach programming tasks differently, and b) these differences can be automatically detected, opening up the possibility that they could be leveraged for pedagogical gain.

Analyzing the process data of students as they complete programming assignments has the potential to provide computing educators with insights into both their students and the processes by which they learn to program. In prior research, we explored the relationship between (a) students’ programming behaviors and course outcomes, and (b) students’ participation within an online social learning environment and course outcomes. In both studies, we developed statistical measures derived from our data that significantly correlate with students’ course grades. Encouraged both by social theories of learning and a desire to improve the accuracy of our statistical models, we explore here the impact of incorporating our predictive measure derived from social behavior into three separate predictive measures derived from programming behaviors. We find that, in combining the measures, we are able to improve the overall predictive power of each measure. This finding affirms the importance of social interaction in the learning process, and provides evidence that predictive models derived from multiple sources of learning process data can provide significantly better predictive power by accounting for multiple factors responsible for student success.

During my first full year as editor-in-chief of ACM Transations on Computing Education (TOCE), the editorial board and I took the following four steps to improve the journal's ability to serve the growing community of computing education researchers: (1) We streamlined the peer review process, (2) we established a new partnership with the ACM Special Interest Group on Computer Science Education Conference, (c) we transitioned to a double-blind review process, and (4) we recruited guest editors for two new special issues that address timely research topics. In this editorial, I present key statistics on TOCE's review process and submissions during the 2016 calendar year, discuss and reflect on the positive steps we took to improve the journal during 2016, and describe steps we will consider in the coming year in order to ensure that ACM TOCE continues on its positive trajectory. These include forging additional partnerships with professional conferences, altering the review criteria to make the journal more welcoming to a broader range of research, especially within the K-12 space, and developing a set of evidence standards for research published in the journal.

In recent years, learning process data have become increasingly easy to collect through computer-based learning environments. This has led to increased interest in the field of learning analytics , which is concerned with leveraging learning process data in order to better understand, and ultimately to improve, teaching and learning. In computing education, the logical place to collect learning process data is through integrated development environments (IDEs), where computing students typically spend large amounts of time working on programming assignments. While the primary purpose of IDEs is to support computer programming, they might also be used as a mechanism for delivering learning interventions designed to enhance student learning. The possibility of using IDEs both to collect learning process data, and to strategically intervene in the learning process, suggests an exciting design space for computing education research: that of IDE-based learning analytics . In order to facilitate the systematic exploration of this design space, we present an IDE-based data analytics process model with four primary activities: (1) Collect data, (2) Analyze data, (3) Design intervention, and (4) Deliver intervention . For each activity, we identify key design dimensions and review relevant computing education literature. To provide guidance on designing effective interventions, we describe four relevant learning theories, and consider their implications for design. Based on our review, we present a call-to-action for future research into IDE-based learning analytics.

Computing education researchers have become increasingly interested in leveraging log data automatically collected within computer programming environments in order to understand students' learning processes and tailor instruction to student needs. While data on students' programming activities has been positively correlated with their learning outcomes, those data tell only part of the story. Another part of the story lies in students' social activities, which, according to social learning theory, can also be predictive of students' learning outcomes. In order to gain further insight into how computing students' learning processes influence their learning outcomes, we present an empirical study that explores the interplay of students' social activities, programming activities, and course outcomes in an early computing course. By analyzing log data collected through a programming environment augmented with a social networking-style activity stream, we found that answers to questions posed through the activity stream were positively correlated with students' ability to make programming progress, and their eventual success in the course. Based on our findings, we present recommendations for the design of pedagogical environments to support a more social programming process.

With the widespread availability of massive amounts of student programming data, we are witnessing a digital gold rush as researchers attempt to make sense of students' programming behaviors. In prior research, we incorporated programming data into a statistical model that accounted for a significant amount of a student's course performance. In a separate line of research, we explored how online social networking tools might be leveraged for pedagogical purposes. Rather than treating our explorations of students' programming and social behaviors as separate research spaces, we are next considering the interplay between social behavior, programming behavior, and course performance. As a first step, we incorporated online social participation into our statistical model of programming behaviors. The outcome was quite promising: we witnessed a 30% increase in our model's effect size. This result would seem to indicate that neither programming behavior nor social behavior alone can fully account for student performance.

Computing educators have become increasingly interested in learning analytics, which involves collecting and analyzing data on students' learning processes and outcomes for the purpose of improving learning and instructional practices. A variety of computer programming environments enable the automated collection of log data on students' programming processes. In addition, log data on students' online social behavior can be easily collected. All of these data can be analyzed alongside data on students' learning outcomes in order to identify correlations between learning processes and outcomes, and ultimately to better tailor instruction to students' needs. This BOF will provide a platform for discussing the emerging field of learning analytics within the context of computing education. The following questions will serve as a starting point for our discussions: (1) What types of data should we be collecting on computing students' (2) How can we best analyze these data in order to gain meaningful insights into students' learning processes? (3) How can we design effective instructional interventions based on the data we collect and analyze?

In order best to support learning analytics in computing education, learning management systems (LMS) should be interfaced with computer programming environments, in which computing students spend much of their time. To this end, we have developed OSBLE+, an outgrowth of the OSBLE (Online Studio- Based Learning Environment) LMS we have been developing over the past eight years. OSBLE+ supports two innovations that make it particularly well-suited to supporting learning analytics in computing education. First, it connects to the Microsoft Visual Studio computer programming environment via a plugin that gives it access to all programming process data within Visual Studio, including edits, compilation attempts, compilation errors, debugging attempts, and run-time exceptions. This enables OSBLE+ to provide students and instructors with a visual analytics environment in which they can explore, compare, and contrast the programming activities of students in the class. Second, the plugin turns Visual Studio into a social programming environment by injecting into it a social media-style activity stream, which enables students (a) to pose and answer programming questions, and (b) to see and explore the programming activities of their peers. Students' social activities within the activity stream are sent to OSBLE+, and are subsequently included as data in the visual analytics environment. OSBLE+ is open source and freely hosted at http://plus.osble.org. While OSBLE+ presently interfaces only with Visual Studio, a plug-in for the Eclipse programming environment is under active development.

In my inaugural editorial as the new editor-in-chief of the ACM Transactions on Computing Education , I take stock of the journal's progress in its first 6 years of existence, and I describe my plans to help the journal maintain its positive trajectory as a viable and vibrant computing education research journal.

Educators stand to benefit from advance predictions of their students' course performance based on learning process data collected in their courses. Indeed, such predictions can help educators not only to identify at-risk students, but also to better tailor their instructional methods. In computing education, at least two different measures, the Error Quotient and Watwin Score, have achieved modest success at predicting student course performance based solely on students' compilation attempts. We hypothesize that one can achieve even greater predictive power by considering students' programming activities more holistically. To that end, we derive the Normalized Programming State Model (NPSM), which characterizes students' programming activity in terms of the dynamically-changing syntactic and semantic correctness of their programs. In an empirical study, the NPSM accounted for 41% of the variance in students' programming assignment grades, and 36% of the variance in students' final course grades. We identify the components of the NPSM that contribute to its explanatory power, and derive a formula capable of predicting students' course programming performance with between 36 and 67 percent accuracy, depending on the quantity of programming process data.

Social learning theory emphasizes the importance of providing learners with opportunities to observe their peers, and to participate actively in a community. Unfortunately, early computing courses tend to emphasize individual programming assignments, which discourage learners from observing and working with their peers. In order to explore the possibility that increased opportunities for social awareness and interaction while working on programming assignments might influence learning outcomes in early computing courses, we are studying the design and use of social networking-style activity streams in such courses. In an empirical study of the use of two types of activity streams in a CS 2 course - one that was part of a learning management system, and one integrated directly into students' programming environment - we found that students who used the integrated stream were twice as socially active; however, social participation in both environments was positively correlated with students' grades. Our results suggest that the use of activity streams as an adjunct to individual programming assignments can positively influence learning; computing instructors would do well to find ways to get their students to participate actively in activity streams during the programming process.

This session will demonstrate best practices for introducing peer instruction and pedagogical code reviews into the computer science classroom. Presenters play the role of instructors on the first day of class. Audience members play the role of students as they collaborate on a variety of sample activities. By providing models of active learning in computing contexts, we seek to motivate instructors to adopt these pedagogies in their classrooms. Sufficient time will be reserved for questions and discussion.

Studio-based learning (SBL), the centerpiece of architecture and fine arts education for over a century, has become increasingly attractive to computing educators. SBL can be conceptualized as an iterative process of solution refinement that relies heavily on shared physical space and design critiques. This session will explore ways in which CS educators can transition aspects of traditional or face-to-face SBL to an online learning environment. We will discuss the potential for online studios, how they might look and function, and how their effectiveness in promoting student learning might be gauged. As more and more CS courses assume an online presence, we seek to understand both the opportunities and the limitations associated with conducting critical studio work in a digital medium.

Given the increasing importance of soft skills in the computing profession, there is good reason to provide students with more opportunities to learn and practice those skills in undergraduate computing courses. Toward that end, we have developed an active learning approach for computing education called the Pedagogical Code Review (PCR). Inspired by the code inspection process used in the software industry, a PCR is a collaborative activity in which a small team of students, led by a trained moderator: (a) walk through segments of each other's programming solutions, (b) check the code against a list of best coding practices, and (c) discuss and log issues that arise. To evaluate the viability and effectiveness of this approach, we conducted a series of four mixed-method empirical studies of various implementations of PCRs in CS1 courses at Washington State University. The first study validated the viability of the PCR activity. Using a quasi-experimental design, the final three studies evaluated two alternative implementations of PCRs— face-to-face and online . Our results provide evidence that PCRs can promote positive attitudinal shifts, and hone skills in critical review, teamwork, and communication. Based on our findings, we present a set of best practices for implementing PCRs.

This BOF is related to the Special Session Transforming the CS Classroom with Studio-Based Learning (SBL). SBL promotes learning in a collaborative context by having students construct, present, review and refine their work with the guidance of peers and teachers. A team of CS educators and education experts have been implementing and evaluating SBL in CS courses over the past five years. The BOF will introduce SBL to the SIGCSE audience, and engage them in a discussion of the potential of, evidence for, and practical advice regarding SBL as an instructional approach that can motivate as well as teach students. Discussions will include "war stories" from teachers who have adopted the approach in their courses and hands-on activities to help participants apply SBL to their courses.

The studio-based learning (SBL) model aims to promote learning in a social and collaborative context by having learners construct, iteratively refine, and critically review design artifacts under the guidance of instructors and disciplinary experts. Recognizing the potential of SBL to foster the development of not only technical design skills, but also increasingly-coveted communication and collaboration skills, we have been adapting the approach for computing education, implementing it in various courses, and evaluating its impact over the past 5 years. This effort now involves 26 computing courses at 15 institutions in seven states. To our knowledge, this represents the most systematic implementation and evaluation of a pedagogy for computing education to date. This special session will introduce SBL to the SIGCSE community and facilitate a discussion and exchange of ideas. In addition to oral and poster presentations of the SBL model and its evaluation results, the session will feature "war stories" from teachers who have applied the model to their courses and hands-on activities to help attendees apply SBL to their courses.

For over a century, studio-based instruction has served as an effective pedagogical model in architecture and fine arts education. Because of its design orientation, human-computer interaction (HCI) education is an excellent venue for studio-based instruction. In an HCI course, we have been exploring a studio-based learning activity called the prototype walkthrough , in which a student project team simulates its evolving user interface prototype while a student audience member acts as a test user. The audience is encouraged to ask questions and provide feedback. We have observed that prototype walkthroughs create excellent conditions for learning about user interface design. In order to better understand the educational value of the activity, we performed a content analysis of a video corpus of 16 prototype walkthroughs held in two HCI courses. We found that the prototype walkthrough discussions were dominated by relevant design issues. Moreover, mirroring the justification behavior of the expert instructor, students justified over 80 percent of their design statements and critiques, with nearly one-quarter of those justifications having a theoretical or empirical basis. Our findings suggest that PWs provide valuable opportunities for students to actively learn HCI design by participating in authentic practice, and provide insight into how such opportunities can be best promoted.

For over a century, studio-based instruction has served as an effective pedagogical model in architecture and fine arts education. Because of its design orientation, human-computer interaction (HCI) education is an excellent venue for studio-based instruction. In an HCI course, we have been exploring a studio-based learning activity called the prototype walkthrough, in which a student project team simulates its evolving user interface prototype while a student audience member acts as a test user. The audience is encouraged to ask questions and provide feedback. We have observed that prototype walkthroughs create excellent conditions for learning about user interface design. In order to better understand the educational value of the activity, we performed a content analysis of a video corpus of 16 prototype walkthroughs held in two undergraduate/graduate HCI courses. We found that the prototype walkthrough discussions were dominated by relevant design issues. Moreover, mirroring the justification behavior of the expert instructor, students justified over 80 percent of their design statements and critiques, with nearly one-quarter of those justifications having a theoretical or empirical basis. These results suggest that prototype walkthroughs can be useful not only in helping to teach HCI design, but also in helping to gauge students' evolving design knowledge.

Given the increased importance of communication, teamwork, and critical thinking skills in the computing profession, we have been exploring studio-based instructional methods, in which students develop solutions and iteratively refine them through critical review by their peers and instructor. We have developed an adaptation of studio-based instruction for computing education called the pedagogical code review (PCR), which is modeled after the code inspection process used in the software industry. Unfortunately, PCRs are time-intensive, making them difficult to implement within a typical computing course. To address this issue, we have developed an online environment that allows PCRs to take place asynchronously outside of class. We conducted an empirical study that compared a CS 1 course with online PCRs against a CS 1 course with face-to-face PCRs. Our study had three key results: (a) in the course with face-to-face PCRs, student attitudes with respect to self-efficacy and peer learning were significantly higher; (b) in the course with face-to-face PCRs, students identified more substantive issues in their reviews; and (c) in the course with face-to-face PCRs, students were generally more positive about the value of PCRs. In light of our findings, we recommend specific ways online PCRs can be better designed.

The blind and visually impaired community is significantly underrepresented in computer science. Students who wish to enter the discipline must overcome significant technological and educational barriers to succeed. In an attempt to help this population, we are engaged in a three-year research project to build an educational infrastructure for blind and visually impaired middle and high school students. Our primary research goal is to begin forging a multi-sensory educational infrastructure for the blind across the United States. We present here two preliminary results from this research: 1) a new auditory programming environment called Sodbeans, a programming language called Hop, and a multi-sensory (sound and touch) curriculum, and 2) an empirical study of our first summer workshop with the blind students. Results show that students reported a significant increase in programming self-efficacy after participating in our camp.

Given the increasing importance of communication, teamwork, and critical thinking skills in the computing profession, we believe there is good reason to provide students with increased opportunities to learn and practice those skills in undergraduate computing courses. Toward that end, we have been exploring instructional methods, which have been successfully employed in architecture and fine arts education for over a century. We have developed an adaptation of instruction for computing education called the pedagogical code review, which is modeled after the code inspection process used in the software industry. To evaluate its effectiveness, we carried out a quasi-experimental comparison of a studio-based CS 1 course with pedagogical code reviews and an identical CS 1 course without pedagogical code reviews. We found no learning outcome differences between the two courses; however, we did observe two interesting attitudinal trends: (a) self-efficacy decreased more in the traditional course than in the course; and (b) peer learning decreased in the traditional course, but increased in the course. Additional questionnaire and interview data provide further evidence of the positive impact of instruction.

Inspired by the formal code inspection process commonly used in the software industry, we have been exploring the use of pedagogical code reviews (PCRs), in which a team of three to four students, led by a trained moderator, (a) walk through segments of each other's programming assignments, (b) check the code against a list of best coding practices, and (c) discuss and log issues that arise. We have found that PCRs not only improve the quality of students' code, but also positively impact students' communication and sense of community. However, implementing PCRs also presents a key logistical challenge: how to make code solutions and review results accessible to team members before, during, and after the team reviews? To address this challenge, we are developing an online environment specifically tailored to support PCRs. Our environment enables students to submit their code solutions; to review team members' code solutions on-line prior to PCRs; to carry out PCRs; and to resubmit their solutions based upon the results of the PCRs. In an empirical evaluation of the environment in a CS1 course, we found that it not only eased the logistics of implementing PCRs, but also improved the organization and efficiency of the PCR process.

Modern day computing jobs are increasingly emphasizing on soft skills such as design, collaboration and communication skills in addition to programming skills. Having students participate in peer reviews is one possible way to develop such soft skills. However, we have observed that low student participation and low review quality can impede the educational value of the peer review process. To address this issue and improve student learning in computer programming courses, we propose to implement and evaluate social comparisons - a method suggested by socio-psychological research.

The iterative design and evaluation of user interface prototypes is central to the user-centered design process. Designers potentially use a variety of prototyping tools and technologies, ranging from simple art supplies to custom prototyping languages and environments. While we know anecdotally that a range of prototyping technologies are used, there exists little empirical research into what technologies designers actually use in practice, and how and why those technologies are used. To explore these issues, we developed an on-line survey, which was completed by 87 user interface researchers and professionals who attended the CHI 2008 conference. Our results not only provide insight into the current state of user interface prototyping in practice; they also suggest preliminary requirements for the next generation of user interface prototyping software.

Advances in modern novice programming environments (NPEs) have made computer programming more accessible to an increasingly large audience. Unfortunately, 35 percent of the top 125 U.S. universities use C or C++, both of which are widely considered to be challenging for novices because they require one to have a firm understanding of computer memory. To make C and C++ more accessible to introductory programming students at these universities, we are developing TransparenC, a NPE that integrates two key advances in modern NPEs-direct manipulation programming and data visualization.

Formal code inspections are employed by teams of professional software engineers to identify software defects and improve the quality of software. After reviewing a piece of code individually, members of an inspection team come together to log the issues they have found, and to find new ones. Within the scope of a multi-institutional research project to adapt, refine, and evaluate studio-based learning methods in computing education, we are developing an adaptation of the formal code inspection called the pedagogical code review for use in lower-division computer science courses. In a pedagogical code review, a group of three to four students, led by a trained moderator, (a) walk through segments of each other's programming assignments, (b) check the code against a list of best coding practices, and (c) discuss and log issues that arise. We implemented pedagogical code inspections in three lab sessions of a CS 1 course. Through an analysis of inspection logs and exit surveys, we compiled evidence that the reviews improved the quality of students' code, stimulated increasingly sophisticated discussions of programming issues and practices, and promoted a sense of community.

Novices face many barriers when learning to program a computer, including the need to learn both a new syntax and a model of computation. By constraining syntax and providing concrete visual representations on which to operate, direct manipulation programming environments can potentially lower these barriers. However, what if the ultimate learning goal of the novice is to be able to program in conventional textual languages, as is the case for introductory computer science students? Can direct manipulation programming environments lower the initial barriers to programming, and, at the same time, facilitate positive transfer to textual programming? To address this question, we designed a new direct manipulation programming interface for novices, and conducted an experimental study to compare the programming processes and outcomes promoted by the direct manipulation interface against those promoted by a textual programming interface. We found that the direct manipulation interface promoted significantly better initial programming outcomes, positive transfer to the textual interface, and significant differences in programming processes. Our results show that direct manipulation interfaces can provide novices with a “way in” to traditional textual programming.

While the demand for college graduates with computing skills continues to rise, such skills no longer equate to mere programming skills. Modern day computing jobs demand design, communication, and collaborative work skills as well. Since traditional instructional methods in computing education tend to focus on programming skills, we believe that a fundamental rethinking of computing education is in order. We are exploring a new "studio-based" pedagogy that actively engages undergraduate students in collaborative, design-oriented learning. Adapted from architectural education, the studio-based instructional model emphasizes learning activities in which students (a) construct personalized solutions to assigned computing problems, and (b) present solutions to their instructors and peers for feedback and discussion within the context of "design crits." We describe and motivate the studio-based approach, review previous efforts to apply it to computer science education, and propose an agenda for multi-institutional research into the design and impact of studio-based instructional models. We invite educators to participate in a community of research and practice to advance studio-based learning in computing education.

Learning computer science is no longer simply a matter of learning computer programming. Indeed, modern day computing jobs demand design, communication and collaborative skills as well. In order to address this need and make computing education more engaging, motivating, and community-oriented, we have been exploring a ldquostudio basedrdquo approach in which students (a) construct computational solutions to problems that have many possible solution strategies, and then (b) present their solutions to their peers and instructors for feedback and discussion. In the fall of 2007, we implemented this approach in a pre-CS1course by requiring students to present their solutions to five course programming projects in face-to-face review sessions. Interview and observational data collected in this course pointed out several practical and logistical problems surrounding the face-to-face review sessions that diminished their educational effectiveness. To address these problems, we describe the preliminary design of a novel asynchronous Web-based project review system to support and augment face-to-face studio-based review sessions.

Because they are easy to create and modify, low fidelity prototypes are commonly used in early evaluations of user interface designs. Designers typically use either pen-and-paper or various computer-based tools to create and test low fidelity prototypes. However, our informal analyses of these technologies indicate that they do not optimally support the two key, complementary tasks of (a) prototype creation and (b) wizard-of-oz testing. To address this problem, we have developed WOZ PRO, a pen-based software environment for the quick and easy creation and testing of low fidelity user interface prototypes. When designing interface screens in WOZ PRO, designers can easily (a) propagate a design change to other related screens, and (b) specify the set of screens that are reachable from a given screen. In a wizard-of-oz test, WOZ PRO aims to reduce the cognitive load on the wizard by allowing navigation only to those next screens that are valid. An experimental comparison of WOZ PRO and pen-and-paper provides evidence of WOZ PROpsilas potential, and insight into how its design might be improved.

Empirical studies of novice programming typically rely on code solutions or test responses as the basis of their analyses. While such data can provide insight into novice programming knowledge, they say little about the programming processes in which novices engage. For those interested in improving novice programming environments, a key research question arises: How can we collect and analyze data on novice programming that will enable us (a) to analyze and compare the programming processes promoted by alternative novice programming environments, and (b) ultimately to build better novice programming environments? To address this question, we have collected a large video corpus of novices as they construct code solutions in various versions of ALVIS Live! [17], a novice programming environment. Through detailed post-hoc analyses of our video corpus, we have developed a methodology for compiling the moment-by-moment evolution of novice code solutions. Based on an analysis of a model code solution's key semantic components, our methodology enables researchers to document, on a second-by-second basis, (a) what part of a code solution a programmer is focusing on, and (b) where the semantic feedback provided by the programming environment is helping. Although it is time and labor intensive, our methodology provides researchers with a standard set of data and representations for comparing the programming processes promoted by alternative programming environments.

Novices face many barriers when learning to program, including the need to learn both a new syntax and a model of computation. By constraining syntax and providing concrete visual representations on which to operate, direct manipulation programming environments can potentially lower these barriers. However, what if the learning goal of the novice is to be able ultimately to program in conventional textual languages, as is the case for introductory computer science students? Can direct manipulation programming environments lower the barriers to programming, and, at the same time, promote positive transfer to textual programming? To address this question, we designed a new direct manipulation programming interface for ALVIS Live!, a novice programming environment. We then conducted an experimental study that compared the programming outcomes promoted by the new direct manipulation interface to those promoted by ALVIS Live!’s textual programming interface. We found that the direct manipulation interface not only led to significantly better initial programming outcomes, but also to significant positive transfer to the textual interface. Our results show that direct manipulation interfaces can provide novices with a “way in” to traditional textual programming.

Pedagogical algorithm visualization technology aims to assist learners in understanding the dynamic behavior of computer algorithms. A key trend in past experimental studies is that learners benefit most when they are actively engaged with algorithm visualization technology. Inspired by this trend, we are exploring the pedagogical value of a novel active learning activity-the Studio Experience-within the context of an introductory CS1 unit on algorithmic problem-solving. In a Studio Experience, student pairs are given algorithm problems, e.g., design two alternative algorithms that reverse the values in a list. They are tasked both with constructing algorithmic solutions and accompanying visualizations, and with presenting their visualizations for feedback and discussion in a session modeled after an architectural design crit. Through an observational study of studio experience sessions in which students used two alternative forms of visualization technology-art supplies and a computer-based tool-we gained insight into (a) the processes by which students construct visual presentations of algorithms, (b) the characteristics of their visual presentations; (c) the nature of conversations mediated by visual algorithmic solutions; and (d) the kind of visualization technology that best supports these activities. Based on our results, we suggest improvements to the approach, and propose an agenda for future empirical studies.

Pedagogical algorithm visualization systems produce graphical representations that aim to assist learners in understanding the dynamic behavior of computer algorithms. In order to foster active learning, educators have explored algorithm visualization systems that empower learners to construct their own visualizations of algorithms under study. Notably, these systems support a similar development model in which coding the algorithm is temporally distinct from viewing and interacting with the resulting visualization. To explore the benefits of narrowing the gap between coding an algorithm and viewing its visualization, we have implemented "What You See Is What You Code", a novel, "radically dynamic" development model to facilitate learner-constructed algorithm visualizations. In this model, the line of algorithm code currently being edited is reevaluated on every edit, leading to the dynamic update of an accompanying visualization of the algorithm. Analysis of usability studies involving introductory computer science students suggests that the immediacy of the model's feedback can help novices to quickly identify and correct programming errors, and ultimately to understand their code's execution.

Algorithm visualizations graphically illustrate how algorithms work. In prior ethnographic studies of a computer science course in which students were required to construct and present their own algorithm visualizations, we observed that visualizations based on storylines tended to stimulate increased audience interest and involvement. This observation, coupled with the empirical research that substantiates the value of stories as mnemonic devices, raises an interesting research question: Do visualizations with storylines actually help students remember the procedural behavior of an algorithm better than visualizations that do not involve storylines? To investigate this question, we conducted an experimental study that compared the memorability of algorithm descriptions involving differing degrees of spatial and verbal embellishment. The study failed to detect significant differences. We reflect on our lack of significant results, and suggest two alternative paths for future research into the value of story-based algorithm visualization.

The ALgorithm VIsualization Storyboarder (ALVIS) enables beginning and intermediate computer science students both to construct "low fidelity" (rough, unpolished) visualizations of the algorithms under study, and to present those visualizations to their peers and instructor for feedback and discussion. The original ALVIS software was a frail research prototype. We are using an iterative, user-centered design process to develop a public domain version of the software suitable for use in a "studio-based" computer science course. Through a series of empirical studies of both low and high fidelity prototypes of ALVIS, we have reevaluated the original design of ALVIS, and are presently in the process of making three key design changes: (a) the underlying scripting language is being redesigned for simplicity and ease of learning; (b) the interface for creating and placing graphical variables and spatial structures is being simplified and streamlined for ease of use; and (c) new features are being added that explicitly support storytelling.

This paper reports an empirical study comparing the roles of representations in supporting face to face versus distance collaborative learning. findings indicate that while face to face collaborators primarily use knowledge representations to summarize verbal reasoning, distributed collaborators also use knowledge representations to propose ideas before they are discussed, particularly ideas supported by the categorical affordances of the knowledge representation medium. The work suggests that designers of online learning environments should support fluid crosstalk between multiple representations, because online discussions will be distributed across all available representations.