Lack of diversity and high dropout rates among underrepresented students plague the CS discipline. We developed, administered, and validated survey scales measuring social factors that impact the retention and graduation of under-represented CS undergrads at two institutions. Results revealed significant differences between students who identify as men vs. women in terms of computing identity and confidence, and between black and non-black students in terms of familiarity with future opportunities.

Understanding the thought processes of students as they progress from initial (incorrect) answers toward correct answers is a challenge for instructors, both in this pandemic and beyond. This paper presents a general network visualization learning analytics system that helps instructors to view a sequence of answers input by students in a way that makes student learning progressions apparent. The system allows instructors to study individual and group learning at various levels of granularity. The paper illustrates how the visualization system is employed to analyze student responses collected through an intervention. The intervention is BeginToReason, an online tool that helps students learn and use symbolic reasoning-reasoning about code behavior through abstract values instead of concrete inputs. The specific focus is analysis of tool-collected student responses as they perform reasoning activities on code involving conditional statements. Student learning is analyzed using the visualization system and a post-test. Visual analytics highlights include instances where students producing one set of incorrect answers initially perform better than a different set and instances where student thought processes do not cluster well. Post-test data analysis provides a measure of student ability to apply what they have learned and their holistic understanding.

To develop code that meets its specification and is verifiably correct, such as in a software engineering course, students must be able to understand formal contracts and annotate their code with assertions such as loop invariants. To assist in developing suitable instructor and automated tool interventions, this research aims to go beyond simple pre- and post-conditions and gain insight into student learning of loop invariants involving objects. As students develop suitable loop invariants for given code with the aid of an online system backed by a verification engine, each student attempt, either correct or incorrect, was collected and analyzed automatically, and catalogued using an iterative process to capture common difficulties. Students were also asked to explain their thought process in arriving at their answer for each submission. The collected explanations were analyzed manually and found to be useful to assess their level of understanding as well as to extract actionable information for instructors and automated tutoring systems. Qualitative conclusions include the impact of the medium.

We present results in automating the analysis of student verbalizations in online learning environments, using an existing online tool designed to teach students to reason analytically about code as an example. The new extension captures "think-aloud'' data as students work through code reasoning activities. The data is recorded and transcribed automatically and used as input to a natural language processing / machine learning system designed to identify specific student attitudes (e.g., uncertain), behaviors (e.g., guessing), and difficulties (e.g., concept misunderstandings). We present the design and implementation of the tool, an analysis of its transcription accuracy, and an evaluation of its utility in identifying characteristics of student learning.

Developing high quality pedagogical materials and techniques is a challenging but important task. We leverage prior work identifying student misconceptions and difficulties in introductory computing courses to design misconception-based feedback (MBF) to address these difficulties. In MBF, peers working in pairs use prompts to guide their discussion of a recently completed coding assignment. A human autograder group (HAG) simulates the behavior of a typical autograder program, supplying only test cases and their correct outputs, allowing us to factor out the effect of the medium (computer vs. human). Participants completed conceptual pre-tests and post-tests that asked them to explain their reasoning, and we captured screen and audio recordings of the sessions. Our mixed-methods analysis looked for statistical differences in frequency counts of misconceptions and qualitatively analyzed audio/video data and language used in pre/post-test written responses to look for explanations of these differences. We use a mixed-methods approach: looking for statistical differences in frequency counts of misconceptions and qualitatively analyzing audio/video data and language used in pre/post-test written responses for explanations of these differences. Significant benefits of MBF were seen on questions that required students to comprehend differences between pass-by-value and pass-by-reference. Not only did these questions show a greater reduction in misconceptions for the MBF group than the HAG group, but the qualitative analysis provided evidence that the students' improvement in language and understanding of the concepts from pre-test to post-test was directly tied to the MBF intervention. This study presents a promising, low resource technique to address misconceptions about several important computer science concepts.

This working group asserts that Program Comprehension (PC) plays a critical part in the writing process. For example, this abstract is written from a basic draft that we have edited and revised until it clearly presents our idea. Similarly, a program is written in an incremental manner, with each step being tested, debugged and extended until the program achieves its goal. Novice programmers should develop their program comprehension as they learn to code, so that they are able to read and reason about code while they are writing it. To foster such competencies our group has identified two main goals: (1) to collect and define learning activities that explicitly cover key components of program comprehension and (2) to define possible learning trajectories that will guide teachers using those learning activities in their CS0/CS1 or K-12 courses.

Understanding student thinking and identifying student misconceptions are important precursors to developing high quality pedagogical materials and approaches. Prior work has used conceptual assessment surveys and task-based interviews to pursue this knowledge, typically having students evaluate existing code or predict or explain code behavior. However, features of student thinking captured under these conditions may differ from features of student thinking that occurs "in the wild." We present the results of a study conducted with 10 introductory CS students at a large, engineering-focused US university. In this work, students were asked to "Please, think aloud" as they interacted in a development environment, attempting to reverse engineer a solution to a defined task based on a provided executable. We captured and analyzed video recordings of the screen and audio recordings of their utterances to characterize their actions, current task, the relevant CS concept involved, current problem-solving phase, and expressed level of certainty. We also took note of the nature of their uncertainties and if and how these uncertainties were resolved. We found that students showed uncertainty regardless of success at task completion. Students who were successful at the task engaged in live experimentation to address these uncertainties in their thinking and coding. However, the ability to produce a program that behaved correctly did not guarantee that the students fully grasped the underlying concepts: some students submitted erroneous code that coincidentally worked, while expressing doubts about the correctness of their implementation.

In computing, concurrency relates to the situation in which different processes or threads may execute at the same time or in an interleaved manner. Many real-world computing systems utilize concurrency, but concurrency-related concepts have proven difficult for students. With an informed understanding of student reasoning in this context, instructors will be better able to create appropriate pedagogical materials and aid student learning. The research questions we seek to address are: What is the nature of undergraduate computer science students' conceptions of concurrency? and How does student reasoning about concurrency differ across the levels of the undergraduate computing program? To study student conceptions of concurrency, we used Lewandowski et al.'s "Concert Tickets" problem, in which multiple vendors sell tickets to a concert and students are asked to provide a natural language response that identifies problems that may occur and to provide possible solutions. We present an analysis of sixty student responses drawn from all four years of the undergraduate CS program at a large, public university in the United States. The analysis utilizes a qualitative open coding approach to uncover common features in student solutions. Most students identified the concurrency related problem of double-booking and did so uniformly across student year. We performed a thematic analysis of student responses about double-booking and found emerging clusters around language for access control, speed, visualization, human protocols, and added states. Knowledge of these student conceptions can help instructors to construct pedagogical materials that build on the strengths and prior knowledge that students bring with them at each level and to address troublesome aspects of these concepts.

You develop a plan for testing the prototype for a new learning strategy in your class or across institutions. How can you ensure that your plan is clearly understood by reviewers and the managing NSF program officer? What goes through the reviewer's mind once a proposal is submitted? What prompts one proposal to be recommended for funding but another declined? Close examination of the panel review process can inform proposal writing and ensure that reviewers will understand an idea, identify its merit, and value a PI's vision of how the work will broaden participation in STEM education. This workshop steps through the NSF proposal review process from submission of proposal to award or decline, touching on NSF intellectual merit and broader impact criteria, mapping the project pipeline to appropriate evaluation. Participants gain insight into writing a good review and improving one's own proposal writing. For further information and travel support see: https://people.cs.clemson.edu/~etkraem/UPCSEd/. Laptops recommended.

Using freely available online automated reasoning tools, we will demonstrate a sequence of engaging reasoning activities that are suitable to introduce beginning programmers and software engineering students to reason logically and symbolically about code. The automated tools have an underlying verification engine that makes it possible for the tool to offer activities and directed logical feedback not possible with typical development environments. The tools have been used in undergraduate classrooms for multiple years by well over a thousand students. The imperative language used by the tool is integrated with the underlying verification engine, and because it closely resembles many commercial languages, it presents little barrier to student usage. A comprehensive activity-based "Reason with Components" tool takes 5-10 minutes of instructor introduction and allows student exploration of contracts, objects, loops, recursion, and reusable concepts. Multiple versions of "Begin to Reason" tools are designed to help students learn the basics of code tracing in intro CS courses "on their own". Students and instructors can create new activities and can fine-tune the existing activities to their specific needs.

In this BoF we discuss the tenets of culturally responsive computer science and how teachers, professors and providers of professional development can include culturally responsive perspectives in their classes. In contrast to other academic fields, which typically include rigid curricular tracks ostensibly based on academic performance, talent, or ability that pose structural barriers to access to rigorous academic instruction for underrepresented students, the field of computer science education is explicitly focused on broadening participation, as evidenced by the SIGCSE community's consistent emphasis on equitable representation. Culturally responsive computing (CRC) is founded on culturally responsive teaching (CRT) and on CRT's three tenets: asset building (in contrast to deficit approaches), reflection, and connectedness. CRC frames these tenets for the specifics of computing education. CRC's tenet that all students are capable of digital innovation should drive teachers' interactions and relationships with students. CRC also requires that teachers be continually reflective about their privilege and constraints and how those are connected with our worldviews. This topic is significant because teachers must be connected to their students in non-traditional ways that prize diversity as an asset to innovation. The participants are expected to include professors, lecturers, high school teachers and industry experts who are interested in employing culturally responsive computing approaches in their own teaching and professional development activities. A major goal of the BoF is to establish connections among the participants to promote the sharing of resources and best practices.

You develop the prototype for a new learning strategy, and want to test it in class or across institutions. You identify an NSF program that supports proposals for the idea, and then what? What goes through the minds of reviewers once a proposal is submitted? What prompts one proposal to be recommended for funding while another is declined? Close examination of the panel review process can inform proposal writing and ensure that reviewers will understand a PI's idea, identify its merit, and value a PI's vision of how the work will broaden participation in STEM education. This workshop steps through the NSF proposal review process from submission of proposal to award or decline, touching on elements of a good review, NSF intellectual merit and broader impact criteria, elements of a good proposal, assessment and evaluation, and volunteering to review proposals. Participants gain insight into writing a good review and improving one's own proposal writing. The interactive workshop leads participants through each topic by introducing related issues, engaging participants in group exercises designed to explore and share their understanding of the issues, and providing "expert" opinion on these issues. Examples include funded and non-funded projects and a Top Ten List of Do's and Don'ts. For further information see: https://people.cs.clemson.edu/~etkraem/UPCSEd/

This SIGCSE special session provides an opportunity for new researchers in CS education to learn the elements of successful computing education research of different types through a series of exemplar projects. Specifically, this session reports on the findings and example, successful CS education research projects that were discussed and presented at ICER 2016 UP (Understanding and Propagating) CS Ed Research Workshop, sponsored by the National Science Foundation. One goal of the session is to provide a way for proposers of computing education research to ensure that they have well identified education research questions and evaluation mechanisms that are appropriate for the proposal (exploratory vs. design & implementation) according to the Department of Education guidelines. The ICER Workshop was designed to focus exactly on this goal and report to the community.

We conducted an observational case study of pair programmers and solo programmers in a challenging, time-limited programming exam in a split level course at the University of Georgia during the spring semester of 2013. Although we observed no significant difference in performance on the exam, we found that pair programming promoted the early application of critical thinking in the process of solving a challenging programming task. Specifically, the pair programmers were found to conduct more critical thinking and to ask important questions in the program design phase while solo programmers postponed the same required thinking activity and associated questions until the testing and debugging phase. We attribute the solo programmer's failure to delve into the difficult questions to the phenomenon of "bounded rationality" and propose that the social effects of pair programming serve to mitigate this phenomenon with the result that the pair programmers' behaviors better align with good software engineering practices and promote the acquisition of expertise.

No abstract available.

We conducted a quasi-experimental study of pair programming versus solo programming, in the context of a required, four-hour, intermediate level-course that focuses on the acquisition of a second language (C/C++). We found pair programming served as a protective factor for the retention of female students and less experienced students. Our findings support those of prior studies that suggested that pair programming may improve retention for female students and also suggest that pair programming strategies may help to shorten the long pre-requisite chain typically found in CS programs, reduce time to degree, and promote student productivity during undergraduate studies. We also found that solo programmers relied more heavily on TA assistance, which implies that resource-constrained departments might better meet student needs or handle larger groups of students with existing resources through the use of pair programming.

This paper presents a semi-automatic Web service composition approach. This approach ranks all available candidate Web service operations based on semantic annotations and suggests service operations to a human designer during the process of Web service composition. The ranking scores are based on data mediation, functionality and formal service specifications. A formal graph model, an IODAG, is defined to formalize an input/output schema of a Web service operation. Three data mediation algorithms are developed to handle the data heterogeneities arising during Web service composition. The data mediation algorithms analyze the schemas of the inputs and outputs of service operations and consider the structures of the schemas. A typed representation for our data mediation approach, which formalizes the data mediation problem as a subtype-checking problem, is presented. An evaluation is performed to study the effectiveness of different data mediation and service suggestion algorithms used to assist designers composing Web services.

The process of selecting Web services from a large number of potential services available on the Web is a challenging task for users engaged in Web service composition. This work is devoted to resolving this issue by suggesting Web services to the user. Our suggestion algorithm ranks all the available services for the user based on the semantic annotations of a service's inputs, outputs and functionality, as well as pre-conditions and effects, if available. This paper presents multiple algorithms for making suggestions during Web service composition. These algorithms extend traditional Web service discovery algorithms; in particular, they include new techniques for ranking the effectiveness of data mediation.

Concurrent software is notoriously difficult to debug. We investigate the use of UML sequence diagrams to help developers correctly reason about the potential behaviors of buggy concurrent software. We conducted a controlled experiment that compared internal (i.e., "in the head") and external representations for reasoning about multithreaded software. For external representations, participants created multithreaded sequence diagrams. The results of the experiment demonstrate a strong positive effect associated with using external representations. Participants who drew diagrams were significantly more successful at reasoning about the potential behavior of concurrent software. Moreover, participants who produced diagrams with higher levels of detail and with fewer errors tended to achieve greater levels of success. Additionally, this paper contributes an extension to the UML sequence diagram notation for showing behavior of multithreaded software and formal metrics for assessing the complexity of thread interactions.

Empirical studies are important for understanding how well current design methods and notations support development of multi-threaded programs. Unfortunately, concurrency exacerbates an already difficult problem in drawing conclusions from such studies: How to objectively measure the quality of candidate solutions produced by participants in the studies. This paper explores the use of formal modeling and analysis for this purpose. We describe initial findings of a small pilot study to determine if we can objectively differentiate sample candidate solutions with respect to their use of synchronization primitives. To do so, we faithfully model these candidate solutions and various synchronization-related properties in the Finite State Processes (FSP) notation and use the Labeled Transition System Analyzer (LTSA) to analyze the solution models against the properties.

Graduates of computer science degree programs are increasingly being asked to maintain large, multi-threaded software systems; however, the maintenance of such systems is typically not well-covered by software engineering texts or curricula. We conducted a think-aloud study with 15 students in a graduate-level computer science class to discover the strategies that students apply, and to what effect, in performing corrective maintenance on concurrent software. We collected think-aloud and action protocols, and annotated the protocols for a number of behavioral attributes and maintenance strategies. We divided the protocols into groups based on the success of the participant in both diagnosing and correcting the failure. We evaluated these groups for statistically significant differences in these attributes and strategies.

While the sources of complexity in the initial design and verification of multi-threaded software systems are well-documented, less is known of the issues specific to the maintenance of these systems. The literature contains a number of observational studies of programmers performing maintenance, conducted in the context of sequential software and designed to investigate the factors and behaviors that lead to success. To help fill the gap in knowledge in the area of concurrent software maintenance, we conducted a study that refines the findings of two prior studies, those of Littman et al. and of Vessey, to address issues and obstacles that arise in the understanding of concurrent software. We validated these refinements by observing programmers performing corrective maintenance on a small but complex multi-threaded server program.

It is generally accepted that concurrency can be difficult for students to reason about. While some studies provide insight into the nature of these difficulties [6], work remains to be done in understanding the aspects of learning about concurrency that are most difficult, and in developing approaches to dealing with this problem. We have conducted instructor interviews and an observational study of students, identified several key difficulties that students encounter, and developed a diagram that we believe will be an aid to understanding and problem-solving. We present the diagram and results of an initial user evaluation.

The field of program comprehension is characterized by both the continuing development of new tools and techniques and the adaptation of existing techniques to address program comprehension needs for new software development and maintenance scenarios. The adoption of these techniques and tools in industry requires proper experimentation to assess the advantages and disadvantages of each technique or tool and to let the practitioners choose the most suitable approach for a specific problem. The objective of this working session is to encourage researchers and practitioners working in the area of program comprehension to join forces to design and carry out studies related to program comprehension, including observational studies, controlled experiments, case studies, surveys, and contests, and to develop standards for describing and carrying out such studies in a way that facilitates replication of data and aggregation of the results of related studies.

Programs that use multithreaded concurrency are known to be difficult to design. Moreover, research in computer-science education suggests that concurrency and synchronization concepts are generally difficult to master. It stands to reason that comprehension tasks may be more complex for programs that employ concurrency than for sequential programs. We believe that external representations, specifically refinements to some of the popular UML modeling notations, should aid students in mastering fundamental concurrency/synchronization concepts and should enable practitioners to better comprehend the dynamically evolving nature of the these programs. In this paper, we present our synchronization adorned UML (saUML) sequence diagram notation that highlights aspects of thread interactions and describe an empirical study of whether these diagrams, as opposed to purely textual representations, help students to better understand concurrent executions and concurrency concepts, as measured by their ability to answer questions about a particular execution of a multi-threaded system. A statistically significant benefit was found from the study.

The VizEval Suite is an environment designed to support experimentation with and evaluation of program visualization attributes that affect the user's ability to grasp essential concepts. In this paper, we describe the VizEval Suite and an initial experiment conducted both as a test-bed of the VizEval Suite and to study how perceptual/cognitive characteristics of the visualization affect the users' understanding of the program visualization. VizEval is designed to simplify the creation and analysis of such studies. Our experimental results show that some perceptual/cognitive characteristics that help one task (e.g., detection of critical information) may harm another (e.g., localization of critical items), and vice versa. The VizEval software is available for download at http://www.cs.uga.edu/~eileen/VizEval/

UNLABELLED: We present SynView, a simple and generic approach to dynamically visualize multi-species comparative genome data. It is a light-weight application based on the popular and configurable web-based GBrowse framework. It can be used with a variety of databases and provides the user with a high degree of interactivity. The tool is written in Perl and runs on top of the GBrowse framework. It is in use in the PlasmoDB (http://www.PlasmoDB.org) and the CryptoDB (http://www.CryptoDB.org) projects and can be easily integrated into other cross-species comparative genome projects. AVAILABILITY: The program and instructions are freely available at http://www.ApiDB.org/apps/SynView/ CONTACT: [email protected].

SUMMARY: RED-T is a Java application for phylogenetic analysis based on a unique method, RED, that utilizes the ratios of evolutionary distances E(d) to distinguish between alternative evolutionary histories. RED-T allows the user to examine if any given experimental gene shares the same evolutionary history as the designated control gene(s). Moreover, the tool detects any differences in evolutionary history and allows the user to examine comparisons of E(d) for a likely explanation. Lateral gene transfer, which may have a significant influence in organismal evolution is one mechanism that could explain the findings of these RED-T analyses. AVAILABILITY: The application is available online at http://www.arches.uga.edu/~whitman/RED.

Abstract Summary: Gene-finding program evaluation (GFPE) is a set of Java classes for evaluating gene-finding programs. A command-line interface is also provided. Inputs to the program include the sequence data (in FASTA format), annotations of 'actual' sequence features, and annotations of 'predicted' sequence features. Annotation files are in the General Feature Format promoted by the Sanger center. GFPE calculates a number of metrics of accuracy of predictions at three levels: the coding level, the exon level, and the protein level. Availability: The program is freely available at: ftp://[email protected]/molbio/genefind/

Algorithm animation system design has focused primarily on providing advanced graphical capabilities. However, a fundamental mismatch exists between the needs of instructors and the features of existing algorithm animation systems. This mismatch has reduced the rate of adoption of algorithm animation tools. We describe a system, SKA (Support Kit for Animation), whose design is based on an examination of the tasks performed in the process of discussing algorithms and data structures. SKA attempts to support and enhance time-consuming instructional tasks such as tracing and data structure diagram manipulation, while requiring minimal preparation or authoring time.

Interactive Steering can be a valuable tool for understanding and controlling a distributed computation in real-time. With Interactive Steering, the user may change the state of a computation by adjusting application parameters on-the-fly. In our system, we model both the program’s execution and steering actions in terms of transactions. We define a steering transaction as consistent if its vector time is not concurrent with the vector time of any program transaction. That is, consistent steering transactions occur “between ” program transactions, at a point that represents a consistent cut. In this paper, we present an algorithm for verifying the consistency of steering transactions. The algorithm analyzes a record of the program transactions and compares it against the steering transaction; if the time at which the steering transaction was applied is inconsistent, the algorithm generates a vector representing the earliest consistent time at which the steering transaction could have been applied.

MOTIVATION: Computational gene identification plays an important role in genome projects. The approaches used in gene identification programs are often tuned to one particular organism, and accuracy for one organism or class of organism does not necessarily translate to accurate predictions for other organisms. In this paper we evaluate five computer programs on their ability to locate coding regions and to predict gene structure in Neurospora crassa. One of these programs (FFG) was designed specifically for gene-finding in N.crassa, but the model parameters have not yet been fully 'tuned', and the program should thus be viewed as an initial prototype. The other four programs were neither designed nor tuned for N.crassa. RESULTS: We describe the data sets on which the experiments were performed, the approaches employed by the five algorithms: GenScan, HMMGene, GeneMark, Pombe and FFG, the methodology of our evaluation, and the results of the experiments. Our results show that, while none of the programs consistently performs well, overall the GenScan program has the best performance on sensitivity and Missing Exons (ME) while the HMMGene and FFG programs have good performance in locating the exons roughly. Additional work motivated by this study includes the creation of a tool for the automated evaluation of gene-finding programs, the collection of larger and more reliable data sets for N.crassa, parameterization of the model used in FFG to produce a more accurate gene-finding program for this species, and a more in-depth evaluation of the reasons that existing programs generally fail for N.crassa. AVAILABILITY: Data sets, the FFG program source code, and links to the other programs analyzed are available at http://jerry.cs.uga.edu/~wang/genefind.html. CONTACT: [email protected].

The volume of data produced by genome projects, X-ray crystallography, NMR spectroscopy, and electron and confocal microscopy present the bioinformatics community with new challenges for analyzing, understanding, and exchanging this data. At the 1998 Pacific Symposium on Biocomputing, a track entitled 'Molecules to Maps: Tools for Visualization and Interaction in Computational Biology' provided tool developers and users with the opportunity to discuss advances in tools and techniques to assist scientists in evaluating, absorbing, navigating, and correlating this sea of information, through visualization and user interaction. In this paper we present these advances and discuss some of the challenges that remain to be solved.