In this Research Full paper, we present a comparative study of using two pedagogical approaches, Peer Instruction (PI) and Process Oriented Guided Inquiry Learning-like (POGIL-like) in software testing. We discuss our results from multiple studies conducted over a period of four years, both in-person and online, using both PI and POGIL-like in undergraduate software engineering classrooms. Our particular focus is on the topics of unit testing, integration testing, and continuous integration. We compare learning outcomes and shifts in student attitudes compared to pure lecture-based instruction. We included two studies conducted using PI in undergraduate software engineering classes. In these studies, we examined how PI can be used to teach the 3 topics mentioned above. They replicated their own studies over a period of two years and found consistently significant cognitive gains and positive affective shifts in both an in person and online class. For POGIL-like, we included two studies in an undergraduate software engineering class - one in-person, and one online. The intervention design centered around the core tenets of POGIL: no prior information given to students (which is the exact opposite of a flipped classroom), and carefully crafted “models” and “activities”. POGIL-like utilizes “learning cycles”, and activity questions are crafted based on the Explore-Invent-Apply cycle combined with Directed/Convergent/Divergent question types, all aimed at encouraging students to leverage the collective knowledge of the group. These approaches, based on social constructivist concepts, have been well studied in STEM disciplines but are relatively recent arrivals in CS. Our particular contribution is that we analyze both pedagogies in teaching software testing, using results from studies that utilized the same instructor and course materials. We analyze the similarities and differences between both pedagogical approaches, and rate their suitability in teaching software engineering and software testing. Our primary research questions are: •RQl: How do students learn software testing and DevOps using PI and POGIL-like compared to pure lecture format classes? •RQ2: What are the relative advantages and disadvantages for PI and POGIL-like from the instructor's perspective? In each of the studies we analyzed, we collected and classified individual student responses from a pre- and post- course survey for each implementation of the course (lecture, PI or POGIL; in person or online; honors or regular). The pre- and post-surveys each included two components: a technical, content-related survey specifically designed to test student knowledge and synthesis in unit testing, integration testing, and continuous integration, followed by a standard affective instrument aimed at gauging student attitudes and perceptions around the same topics. We adopted within-groups and between-groups Analysis of Variance techniques to determine statistical significance in our correlations. In all 4 studies involving PI and POGIL-like, we found encouraging results showing that students had learned significantly better than pure lecture alone. In this paper, for both PI and POGIL-like, we discuss and compare overall correctness for each group, as well as cognitive and affective gains based on the pre- and post- surveys. We also classify the relative strengths and weaknesses of each pedagogy based on student feedback. We discuss the similarities in these approaches as well as implementation differences, in addition to impediments faced while implementing them, along with ideas to encourage increase more widespread adoption of these evidence-based pedagogies.

In this Research Full paper, we present the results of a replication study in a semester-long, sophomore-level software engineering course utilizing Peer Instruction (PI). PI is an active learning pedagogy with roots in STEM Education. In this study, we examine the relationship between student response data from in-class PI correctness and students' performance on quizzes and exams. We worked with a fully remote, synchronous course offered over Zoom. The study we replicated was with an honors cohort of students with a diversity of undergraduate majors, while we focused on a non-honors course containing computing-related majors. Our intervention design included a flipped-classroom approach for each class session with required readings, reading quizzes, followed by PI in class using online breakout rooms for peer discussion. Our course modules were heavily based on industry practices and knowledge from the workforce, across several varied modules that encompass the complete software development lifecycle, and were as follows: Software Process Models (SPM), Software Architecture (SA), Databases (DB), User Interface/user Experience (UI/UX), Software Testing (ST), and Continuous Integration (CI). Our data points for analysis with fine-grained PI student response data were two-fold: scores from weekly online quizzes, and a summative final exam, administered online through a course management system (CMS), at different points during the semester after the PI sessions. The online quizzes and the online exam were timed, closed book/notes, and conducted during class periods. We analyzed and classified individual student responses before and after each question in each module and attempted to create response patterns for each module. We correlated these response patterns with exam and quiz scores using ANOVA techniques, on a variety of questions including Parson's problems. We report overall correctness on each type of vote, track student response patterns from in-class to quizzes and the exam, and quantify absolute percentages of students that demonstrate longer-term learning from the PI process. Our results show that 58% of students exhibited cognitive gains across all modules during PI sessions. Students who learn in class from PI perform well on the quizzes and the final exam, indicating persistence of the knowledge gained during PI several weeks after the actual sessions. We also found that those who fail to learn from the PI process in the class perform worse on quizzes and the final exam. Our results were consistent across all modules. More significantly, we found PI to be an effective way to teach our software engineering course based on student learning before and after PI, in a completely virtual environment, a result unique to our study. Based on our results, we discuss the implications for software engineering education, both in-person and virtual.

This paper discusses the results of partially replicating and modifying a study performed by Zingaro and Porter examining the relationship between fine grained clicker data from in class Peer Instruction and students' performance in quizzes and exams. Whereas Zingaro and Porter worked with a CS1 course, we worked with a sophomore software engineering course. We report overall answer correctness when students vote before and after PI discussion, track student response patterns from in-class to the quizzes and exam, and quantify absolute percentages of students that demonstrate longer-term learning from the PI process. Our results show that students who learn in class from PI perform well on the quizzes and the final exam, nearly as well as those who understood concepts prior to the classes in which those concepts were taught. We also found that those who fail to learn from the PI process in the class perform worse on quizzes and the final exam. We found PI to be an effective way to teach our software engineering course based on student learning before and after PI, a result unique to our study. Our results were consistent across the different topics in software engineering in which we employed PI.

Peer Instruction (PI) is an active learning pedagogy in which students actively participate in the classroom. There have been several research studies regarding the value of PI in computer science. The present work adds to this body of knowledge by examining outcomes from an undergraduate software engineering course with specific focus on the effects of PI on student learning in the topics of unit testing, integration testing and continuous integration. We find encouraging increases in levels of success as measured through a cognitive pre- and post-course survey for those topics. This work also documents and hypothesizes reasons for the cognitive gains from PI, as well as student attitudes towards PI.

End-user programmers are increasingly relying on Web authoring environments to create Web applications. Although often consisting primarily of Web pages, such applications are increasingly going further, harnessing the content available on the Web through "programs" that query other Web applications for information to drive other tasks. Unfortunately, errors can be pervasive in Web applications, impacting their dependability. This paper reports the results of an exploratory study of end-user Web application developers, performed with the aim of exposing prevalent classes of errors. The results suggest that end-users struggle the most with the identification and manipulation of variables when structuring requests to obtain data from other Web sites. To address this problem, we present a family of techniques that help end user programmers perform this task, reducing possible sources of error. The techniques focus on simplification and characterization of the data that end-users must analyze while developing their Web applications. We report the results of an empirical study in which these techniques are applied to several popular Web sites. Our results reveal several potential benefits for end-users who wish to "engineer" dependable Web applications.