The surge in scientific submissions has placed increasing strain on the traditional peer-review process, prompting the exploration of large language models (LLMs) for automated review generation. While LLMs demonstrate competence in producing structured and coherent feedback, their capacity for critical reasoning, contextual grounding, and quality sensitivity remains limited. To systematically evaluate these aspects, we propose a comprehensive evaluation framework that integrates semantic similarity analysis and structured knowledge graph metrics to assess LLM-generated reviews against human-written counterparts. We construct a large-scale benchmark of 1,683 papers and 6,495 expert reviews from ICLR and NeurIPS in multiple years, and generate reviews using five LLMs. Our findings show that LLMs perform well in descriptive and affirmational content, capturing the main contributions and methodologies of the original work, with GPT-4o highlighted as an illustrative example, generating 15.74% more entities than human reviewers in the strengths section of good papers in ICLR 2025. However, they consistently underperform in identifying weaknesses, raising substantive questions, and adjusting feedback based on paper quality. GPT-4o produces 59.42% fewer entities than real reviewers in the weaknesses and increases node count by only 5.7% from good to weak papers, compared to 50% in human reviews. Similar trends are observed across all conferences, years, and models, providing empirical foundations for understanding the merits and defects of LLM-generated reviews and informing the development of future LLM-assisted reviewing tools. Data, code, and more detailed results are publicly available at https://github.com/RichardLRC/Peer-Review.

A technical interview is the culmination of the recruiting process for hiring software engineers in the tech industry. Many well-known companies, including Amazon, Meta (formerly Facebook), Alphabet (Google), and Microsoft, use it to filter candidates. However, the drawbacks of technical interviews are well-documented, including their lack of real-world relevance, bias towards newer developers, demanding time commitment, and potential to induce unnecessary anxiety and frustration. De-spite these criticisms, there is no clear indication that the industry will alter the format of technical interviews in the near future. To assist student developers in preparing for these challenges, we conducted a quantitative analysis using over 300,000 user profiles from LeetCode, arguably the most popular online platform for preparing software development candidates for interviews. Our analysis aims to provide developers with insights into the effort required to prepare for technical interviews, especially in terms of solving programming questions, to secure a position at a renowned company.

In addressing the need for test case generation in software projects and the validation and repair processes, various algorithms and AI models are increasingly being applied with novel approaches. On the other hand, despite the established effectiveness of the Test-Driven Development (TDD) approach in testing and development, there is still a lack of research examining the impact of human-machine interaction on software validation and coding. This paper introduces a tool, the test-skeleton generator, which utilizes an OpenAI model to generate test skeletons. These skele-tons include test names, signatures, and scenario descriptions, omitting the actual test bodies. To explore the implications of this tool, an empirical experiment involving student participation was conducted to assess the conversion of test skeletons into functional tests with human-machine interaction. The study reveals significant insights, indicating that human-machine interaction plays a crucial role in shaping both the testing and programming phases, encouraging students to prioritize writing tests before modifying source code. Teams adopting this approach demonstrate a tendency to produce more tests, leading to higher code coverage. Additionally, our research underscores the growing potential of AI language models to generate tests that closely resemble those written by human developers. Notably, human-machine interaction has proven its significant positive impact on the validation and repair process of AI -generated tests.

This research paper describes how pattern recognition can be used to validate student-assigned labels and improve a training dataset natural language processing. In our preceding research, we attempted to enhance peer assessment by harnessing the power of natural language processing (NLP) and machine learning (ML) to critically evaluate the substance and quality of peer-review comments. This required training data, which we had students create by labeling review comments they had received from peers on various aspects of their work. Our previous paper delineated strategies for the automatic validation of labels (“tags”) applied by students, aiming to enhance the reliability of the data fueling our ML algorithms. This paper examines the metrics introduced in our previous work, and studies how effective they are in the evaluation of actual labels assigned by students.

This is a research-to-practice full paper. Several research studies indicate that women who have opted into a computing career path must regularly contend with negative stereotypes about their technical abilities. These stereotypes are often cited as contributing factors to the underrepresentation of women in computing. To counter these stereotypes and enhance female participation in computer science, numerous interventions have been designed. However, most existing research tends to rely on anecdotal evidence and questionnaires to study these stereotypes. In contrast, our study collected data from over 900 students over a span of eight years and adopted a comprehensive quantitative approach to examine these stereotypes about female students. We utilized pre-class GitHub contribution metrics to evaluate students' programming experience and an array of in-class grading items to measure students' performance. Additionally, we mined the project repositories' git logs to gain insights into students' contributions to team projects. Our investigation began by probing whether there was a notable difference in the technical backgrounds or preparedness between female and male students. The results indicated that males tended to be better prepared. Next, we explored potential disparities in class performance between the two genders. Our findings revealed that males and females each excelled in different areas. We were also interested in discerning if female and male students contributed equally to team projects; our analysis affirmed that the contributions were comparable between the two groups. If allowed to choose their teammates, we examined whether they showed a preference for single-gender teams or mixed-gender teams. Our conclusions indicated no marked preference. This paper aims to augment the body of research on computing education by assisting educators in gaining a better understanding of female students in the class. Moreover, it tests the stereotypes by comparing them with empirical results.

This research paper describes an interactive system using large language models and prompt engineering to generate rubrics. Rubrics have long been employed to ensure a grading system that is both equitable and consistent. In practice, generating rubrics could be challenging for instructors for many reasons (e.g., a tight course schedule, limited resources, and varying materials for different projects in the same course), which urges the need to generate the rubric automatically. To the best of our knowledge, little research has been performed on generating rubrics. In this work, we present a novel system based on Large Language Models (LLMs) and Prompt Engineering to help instructors generate rubric items interactively based on course materials, as well as assess the student's work using these rubrics to give timely feedback automatically. In this system, we applied several LLMs (e.g., GPT4, Llama, Falcon, and Hermes) to generate both rubric and feedback using this process: 1) a set of text chunks are initially generated from the textual materials (these textual materials may from various sources), then LDA (Latent Dirichlet Allocation) is applied to extract a set of keywords from the preprocessed text chunks for rubric generation; 2) a web page was designed to let the instructor choose if the keywords from the set are adequate as rubric words; 3) the rubric items are generated by LLMs from the rubric words. In our experiments, a total number of 1017 documents (including the syllabus, the course website, the requirement of projects, the students' works, and the instructors' feedback) were used to build the corpus to generate the rubric-related keywords. Three users (including one instructor and two teaching assistants) participated in generating the rubric interactively using the webpage. The results of experiments show that the interactively generated rubrics from the LLM-instructor system can achieve a level similar to manually created rubrics. We utilized different prompts to let the LLMs generate feedback for the student's work, based on the generated rubrics. Our study shows that generating automatic rubrics and feedback for student project reports is feasible, yet it also identifies significant challenges that future research needs to address.

In modern software engineering education, team formation is crucial for mimicking real-world collaborative scenarios and boosting project-based learning outcomes. This paper introduces a simple, innovative, and universally adaptable method for forming student teams within a software engineering class. We utilize publicly available pre-class GitHub metrics as our input variables (e.g., number of commits, pull requests, code size, etc.). For team formation, the constrained k-means algorithm is employed. This algorithm embraces domain-specific constraints, ensuring the resulting teams not only resonate with the inherent data clusters but also meet educational requirements. Preliminary results suggest that our methodology yields teams with a harmonious blend of skills, experiences, and collaborative potentials, thereby setting the stage for enhanced project success and enriched learning experiences. Quantitative analyses show that teams formed via our approach outperform both randomly assembled teams and student self-selected teams concerning project grades. Moreover, teams created using our method also display a reduced standard deviation in grades, suggesting a more consistent performance across the board.

This study presents a comparative analysis of contributions to GitHub by students before and after participating in a Software Engineering class based on Open Source Software (OSS). The primary objective is to understand the influence of formal software engineering education on students' engagement in OSS projects, as reflected in their GitHub activities. The research addresses two key questions. Firstly, it examines how GitHub contributions change before and after the class. The corresponding hypothesis posits that students' average GitHub contributions will exhibit a distinct pattern post-class compared to pre-class. Additionally, the study explores the potential association between students' academic performance in the class and their level of GitHub contributions after the class. The strength and direction of the potential association are quantified using the Spearman correlation coefficient, considering the potential non-linear nature of the data. This analysis uses data from over 1000 students across more than 10 years, encompassing their GitHub contribution data over multiple timeframes and their grades in the class. The study employs a combination of statistical methods, including paired tests and correlation analysis, to explore these dynamics. While causality cannot be established due to the absence of a control group, the findings offer valuable insights into the correlation between academic engagement and practical contributions in the realm of OSS development. This research contributes to the understanding of how theoretical software engineering education might relate to practical application and engagement in real-world projects.

Engaging students in the peer review process has been recognized as a valuable educational tool. It not only nurtures a collaborative learning environment where reviewees receive timely and rich feedback but also enhances the reviewer’s critical thinking skills and encourages reflective self-evaluation. However, a common concern arises when students encounter misaligned or conflicting feedback. Not only can such feedback confuse students; but it can also make it difficult for the instructor to rely on the reviews when assigning a score to the work. Addressing this pressing issue, our paper introduces an innovative, AI-assisted approach that is designed to detect and highlight disagreements within formative feedback. We’ve harnessed extensive data from 170 students, analyzing 15,500 instances of peer feedback from a software development course. By utilizing clustering techniques coupled with sophisticated natural language processing (NLP) models, we transform feedback into distinct feature vectors to pinpoint disagreements. The findings from our study underscore the effectiveness of our approach in enhancing text representations to significantly boost the capability of clustering algorithms in discerning disagreements in feedback. These insights bear implications for educators and software development courses, offering a promising route to streamline and refine the peer review process for the betterment of student learning outcomes.

Group or team projects are an essential component of the software engineering curriculum. Earlier studies have explored how prior programming experience influences students' team project performance and overall class performance in software engineering. However, few studies address the impact of prior programming experience on students' contributions to team projects. Previous work has varied in its definitions of prior programming experience or skill, leading to inconsistent findings. In this study, we collected pre-class GitHub contribution metrics from 237 students (forming 79 teams of three) across two academic years to measure their prior programming experience and skills. We also mined students' project repositories' git logs to collect individual student contributions. A central question revolved around whether students with more substantial prior programming experience were indeed more active contributors to their project teams. Interestingly, our data indicated a positive correlation between prior programming experience and contributions to team projects. We further delved into team dynamics. Specifically, we questioned if teams made up of members with comparable skill levels exhibited a more even distribution of contributions. Contrary to expectations, our findings revealed no association between these two variables. Moreover, we investigated the team configurations that might encourage the rise of "free riders"-students who contributed only minimally. This paper seeks to augment the body of research on computing education and assist educators in understanding how prior programming experience impacts students' contributions in team projects.

The effectiveness of machine learning depends on the quality of the training data. In some fields, the training data is generated from crowdsourced labels. Ensuring the reliability and accuracy of crowd-generated labels is a critical challenge. To address this, we present comprehensive research to develop and implement robust quality control strategies in crowd labeling. We focus on enhancing the effectiveness of feedback and suggestions by evaluating the taggers and the quality of tags they assign by using natural language processing techniques and machine learning. Our approach aims to assign reliability metrics to each tagger and tag, enabling researchers to filter and create machine-learning training datasets from the most reliable annotations. This research addresses this issue by implementing four quality control strategies in the domain of peer assessment and comparing their performance with manual grade scores assigned by professors. The four key quality control strategies encompass identifying taggers who tag too quickly, detecting taggers providing inconsistent labels, uncovering unreliable taggers employing pattern-based tagging, and performing agreement/disagreement analysis for tags. By individually implementing and assessing the impact of each strategy on data alignment with manual grade scores, we ascertain their effectiveness in enhancing the reliability of crowd -generated labels.

Research into predicting students' performance in computer science classes has been conducted globally for over five decades. Numerous metrics, including performance in prior courses, demographic information, and programming experience, have been used to predict success in computer science. Various analytical methods, such as linear regression, decision trees, ensemble methods, and even neural networks, have also been explored. In this study, we investigate whether pre-class GitHub contribution metrics, combined with machine learning techniques, can forecast student performance in a software engineering class. We address two research questions in this paper. Firstly, can pre-class GitHub contribution metrics predict students' performance? Secondly, which machine learning technique is most effective in predicting student performance? We collected data from 802 students over five years and 11 semesters, including pre-class GitHub contribution stats, students' exam grades, project grades, documentation grades, and review writing grades. Eight different machine learning methods were then tested to predict in-class performance using pre-class GitHub contributions. Our results indicate that exam performance can be relatively accurately predicted by machine learning methods. Ensemble methods such as Random Forest, AdaBoost, and XGBoost performed better than other methods. This suggests that pre-class GitHub contribution metrics can be a useful tool for predicting students' performance in software engineering classes, carrying significant implications for educators. This approach can help educators identify at-risk students at the earliest point in the class, enabling early intervention strategies to prevent failure. Our study uniquely utilizes pre-class GitHub contributions, providing a preliminary indication of a student's familiarity with the course material. While prior research has focused on using in-class data to predict student performance, our approach identifies struggling students from the very beginning. We believe this can provide the most beneficial support for students.

What skills does a student need to succeed in a programming class? Ostensibly, previous programming experience may affect a student's performance. Most past studies on this topic use self-reporting questionnaires to query students about their programming experience. This paper presents a novel, unified, and replicable way to measure previous programming experience using students' pre-class GitHub contributions. To our knowledge, we are the first to use GitHub contributions in this way. We conducted a comprehensive statistical study of students in an object-oriented design and development class from 2017 to 2022 (n = 751) to explore the relationships between GitHub contributions (commits, comments, pull requests, etc.) and students' performance on exams, projects, designs, etc. in the class. Several kinds of contributions were shown to have statistically significant correlations with performance in the class. A set of two-samplet -tests demonstrate statistical significance of the difference between the means of some contributions from the high-performing and low-performing groups.

This full research paper shows how machine learning can improve peer assessment by giving students advice on how to write better quality reviews. We trained a model that gives automated feedback by using labeled data produced by students over a period of several semesters. To improve the accuracy of the model, we are working to incorporate active learning (in the machine-learning sense) to direct students to produce training data for situations where the model has the most difficulty making predictions. With the active-learning approach, we expect students to have to do less labeling, so that they can be more attentive and produce more accurate labels. Our results revealed that we are able to cut the amount of labeling effort by half, without loss of reliable training data.

Teamwork skills are much in demand in the workplace, even more so with the growth of Agile methods. This calls for giving Computer Science students more practice in the kinds of team scenarios they will encounter on the job. Key for success are hands-on experience with planning methods, prioritization techniques, time management and organization. This poster shows how the cooperative tracking tool Github Project Boards helps teams strategize development, track progress, distribute work evenly, and facilitate collaboration. It also shows how instructors can use Github Project Boards to visualize and evaluate a team's development process.

Rising enrollments make it difficult for instructors and teaching assistants to give adequate feedback on each student's work. Our course projects require students to submit GitHub pull requests as deliverables for their open-source software (OSS) projects. We have set up a static code analyzer and a continuous integration service on GitHub to help students check different aspects of the code. However, these tools have some limitations. In this paper, we discuss how we bypass the limitations of existing tools by implementing three Internet bots. These bots are either open source or free for OSS projects and can be easily integrated with any GitHub repositories. One-hundred one Computer Science and Computer Engineering masters students participated in our study. The survey results showed that more than 84% of students thought bots can help them to contribute code with better quality. We analyzed 396 pull requests. Results revealed that bots can provide more timely feedback than teaching staff. The Danger Bot is associated with a significant reduction system-specific guideline violations (by 39%), and the Code Climate Bot is associated with a significant 60% decrease of code smells in student contributions. However, we found that the Travis CI Bot did not help student contributions pass automated tests.

Test-driven development (TDD) promises to help students write high-quality code with fewer defects. Although many studies of TDD usage have been conducted in entry-level computer science courses, few have looked at more advanced students doing projects of larger scope, such as contributing to open-source software (OSS). To test the performance of the test-driven approach on OSS-based course projects, we conducted a quasi-experimental controlled study, which lasted for more than one month. Thirty-five masters students participated in our study. They worked on course projects in teams, half of which were assigned to the TDD group (using a test-driven approach), and the rest of which were assigned to the non-TDD group (using the traditional test-last approach). We found that students in the TDD group were able to apply test-driven techniques pragmatically-spending more than 20% of their time on average complying with the test-driven process-throughout the whole project. There were no major differences in the quality of source-code modifications and newly added tests between the TDD group and the non-TDD group; however, the TDD group wrote more tests and achieved significantly higher (12% more) statement coverage.

Rising enrollments make it difficult for instructors and teaching assistants to give adequate feedback on each student's work. In our software engineering course, we have 50-120 students each semester. Our course projects require students to submit GitHub pull requests as deliverables for their open-source software (OSS) projects. We have set up a static code analyzer and a continuous integration service on GitHub to help students check code style and functionality. However, these tools cannot enforce system-specific customized guidelines and do not explicitly display detailed information. In this study, we discuss how we bypass the limitations of existing tools by implementing three Internet bots. The Expertiza Bot can help detect violations of more than 35 system-specific guidelines. The Travis CI Bot can explicitly display instant test execution results on the GitHub pull-request page. The Code Climate Bot can insert pull-request comments to remind students to fix issues detected by the static code analyzer. These bots are either open source or free for OSS projects, and can be easily integrated with GitHub repositories. Our survey results show that more than 70% of students think the advice given by the bots is useful. We tallied the amount of feedback given by the bots and the teaching staff for each GitHub pull request. Results show that bots can provide significantly more feedback (six times more on average) than teaching staff. Bots can also offer more timely feedback than teaching staff and help student contributions avoid more than 33% system-specific guideline violations.

Open-source course projects offer students a glimpse of real-world projects and opportunities to learn about architectural design and coding style. While students often have more difficulties with these projects than with traditional "toy" projects, instructors are also spending excessive time on grading miscellaneous projects. There is an improvising need for means to help students and instructors with their difficulties. This poster presents our work on predicting which course projects are likely to fail at an early stage with machine learning approaches. We collected metadata from 247 course projects in a graduate-level Object-Oriented Design and Development course over the past 5 years, built models to fit the course projects and use the classifier to help instructors to identify potential failing projects, thus to help students to salvage their works. By assuming that the project acceptances are related to the working patterns of project teams, we made innovations of adding temporal-based patterns into the training data, and achieved 86.36% classification accuracy with the addition of those features. We also proved several observations, such as most of the rejected projects are those begun relatively late during the project period, and the projects which modified more files/code does not result in better possibility of being accepted. By contrast, accepted projects tend to deliver a volume of code that is neither very small nor very large, compared to rejected ones. Our results also suggest that setting milestone checkpoints at roughly a week before the submission deadlines would enable more students to succeed in their OSS projects.

Several decades of research have shown peer assessment to be an effective pedagogical approach. Researchers have shown that peer assessment has the potential to provide students more copious, timely and helpful feedback, and also helping reviewers to learn as well. In recent decades, peer assessments in educational settings have increasingly been facilitated by online tools. Some MOOC platforms also rely on aggregated peer-assessment scores to assign grades for each artifact. However, in peer assessment, students can potentially game this process, and thereby harm the validity and reliability of the aggregated scores. Most of instructors assume that the majority of the students' peer assessments are honest since most of the peer assessment is done in double-blind fashion. This assumption only holds in the absence of organized collusion - when no more than a small number of students game the peer assessment and give each other very high scores. This paper identifies two types of collusion that we have observed. They are small-circle collusion and pervasive collusion. Small-circle collusion refers to the behaviors of students who form small circles and give higher peer review grades to each other. Pervasive collusion refers to students assigning top grades to all the submissions they review. We also present our algorithms for detecting these two types of colluders. Our experiments are based on a peer-assessment dataset shared by multiple peer-assessment systems. By removing these colluders' peer assessments, we are able to estimate how much inflation is brought by colluders in educational peer assessment.

In studio-based education, students collaboratively work on projects that allow them to learn aspects of design through experience on authentic artifacts, often for outside clients. In our Object-Oriented Design and Development class, this means work on open-source projects, such as Mozilla Servo, OpenMRS, Sahana Eden, and Apache Ambari, as well as our own Expertiza project. This paper recounts five years of experience awarding extra credit for activity that students engaged in to help fellow students. These activities included doing extra peer reviews of classmates' work, helping other teams with their projects, writing various types of quiz questions, and answering questions on the Piazza message board. The incentives often led to “gaming” behavior, where students engaged in activities simply to earn points, with very little benefit to their fellow students. Consequently, the scoring policy has been changed several times during the five-year period. We show how student behavior changed as the rules changed, how we have enhanced review quality, improved responses and response time on Piazza, and transitioned students away from merely helping with project setup, and toward helping other students with the substance of their projects.

Educational peer assessment has proven to be a powerful approach for providing students timely feedback and allowing them to help and learn from each other. In an educational setting, most peer assessment consists of a single round. The problem with this setting is that, either the authors do not have a chance to update their work, which makes the suggestions from their peers useless, or the author can make changes after receiving the peer reviews, which forecloses using peer review to help assign grades. To address these issues, in our classes we now use two rounds of online review, with a different rubric for each. Our Expertiza peer-review system allows the evaluation rubric to vary by rounds. In the first review round, we present a formative review rubric to the peer reviewers. In the formative rubric, we try to encourage student reviewers to look into details, point out the problems they can find in the author's work, and offer insightful suggestions. After the formative review round, authors have the opportunity to submit an updated version of their artifacts. Next comes a summative peer-review round, using a summative rubric. A summative peer-review rubric focused more on evaluating the quality of the artifact by comparing it against specific benchmarks. In this paper, we discuss the design of the two-round peer-review assignments in a computer-science course and present our observations on student peer-review activity. An analysis of students' peer-assessment responses confirms the effectiveness of this design of peer-review activity.

Peer assessment has proved to be a useful technique in all levels of education. The process of giving and receiving comments can encourage critical thinking and help students learn both from reviewing and being reviewed. Peer assessment generates a large volume of data, especially if done online. Online peer-assessment systems are designed differently and use different schema for their data, which complicates the work of comparing different designs. For example, some systems are based on ranking - reviewers rank the artifacts they are asked to assess, while other systems use rating - reviewers assess a single artifact at a time and score it on various criteria. Comparing these two types of systems, e.g. on rating accuracy, or usefulness of formative feedback, can be challenging because researchers need to learn the design and terminology of each system before analyzing the data. We introduce a Peer-Review Markup Language to provide a common definition of terminology across multiple systems. We are using this markup language to build a data warehouse for data from different systems. We discuss issues raised during this process and our approach to solving them.

Peer review has long been used in education to provide students more timely feedback and allow them to learn from each other's work. In large courses and MOOCs, there is also interest in having students determine, or help determine, their classmates' grades. This requires a way to tell which peer reviewers' scores are credible. This can be done by comparing scores assigned by different reviewers with each other, and with scores that the instructor would have assigned. For this reason, several reputation systems have been designed; but until now, they have not been compared with each other, so we have no information about which performs best. To make the reputation algorithms pluggable for different peer-review system, we are carrying out a project to develop a reputation web service. This paper compares two reputation algorithms, each of which has two versions, and reports on our efforts to make them “pluggable,” so they can easily be adopted by different peer-review systems. Toward this end, we have defined a Peer-Review Markup Language (PRML), which is a generic schema for data sharing among different peer-review systems.

With interest in "flipped classrooms" rapidly growing, CS faculty are looking for ways to "flip" with a reasonable amount of effort and good results, There is no lack of resources or advice; videos and video-recording apps are proliferating, as are articles and web sites devoted to flipping. Only a small fraction of this is specifically targeted at CS, so it is a daunting task to sort through the available material to find what works well in computing. This BoF will bring together educators who have used, and can recommend, resources for flipping to their colleagues.

We report on the experience of teaching an industry-validated course on Artificial Intelligence in Computer Games within the Simulation and Game Design department at a two-year community college during a 16-week semester. The course format used a blended learning just-in-time teaching approach, which included active learning programming exercises and one-on-one student interactions. Moskal's Attitudes Toward Computer Science survey showed a positive and significant increase in students in both interest (W(10) = 25, p = 0.011) and professional (W(10) = 49.5, p = 0.037) constructs. The Felder-Soloman Index of Learning Styles (n = 14) failed to identify any statistically significant differences in learning styles when compared to a four-year CS1 class. In the final class evaluation, 8 out of 13 students (62%) strongly or very strongly preferred the blended learning approach. We validated this course through four semi-structured interviews with game companies. The interview results suggest that companies are strongly favorable to the course content and structure. The results of this work serve as a template that community colleges can adopt for their curriculum.

In most of education, courses are graded based on percentages-a certain percentage is required for each letter grade. Students often see this as a negative, in which they can only lose points, not gain points, and put their class average at risk with each new assignment. This contrasts with the world of online gaming, where they gain “experience points” from each new activity, and their score monotonically increases toward a desired goal. In Fall 2012, the lead author switched to grading by experience points in his Ethics in Computing class. Students earned points for a variety of activities, mainly performing ethical analyses of various issues related to computing, and participating in debates on ethics-related topics. The students appreciated the ability to earn extra points by performing extra activities. But they were less likely to complete analyses after signing up to do them than were students in a traditionally-graded class. At semester's end, the number of peer reviews increased, as students strove to top off their point total. The grade distribution was bimodal, with clusters at both ends (A+ and F). Students' greatest concern was rapid grading turnaround, so they would know where they stood in the class at all times.

Textbook pricing has become a hot-button issue in academe. Allegations are rife that textbook prices are increasing faster than tuition, and that some students spend even more on textbooks than on tuition. The matter is complicated by rapidly changing technology and the expectations that print textbooks may soon be obsolete. This panel brings together two textbook editors from major publishers and two authors with extensive experience in new media to discuss why textbooks are so expensive and how the market is likely to change in the foreseeable future.

This paper reports on an experience in using the inverted-lecture model ("flipping the classroom") in computer architecture. The first author concurrently taught two courses in computer architecture. One of these courses was CSC/ECE 506: Architecture of Parallel Computers, an introductory Graduate-level course, taught via lecture both residentially and distance-ed. The other was the CSC 456: Computer Architecture and Multiprocessing, a senior-level undergraduate course which was "flipped." Students in the inverted-lecture class exhibited high levels of engagement. Their performance on exams was not quite up to the level of the students in the graduate class, but the difference was not wide. From this experience, we offer observations and suggestions about inverted classes in general.

In this paper we propose a new word-order based graph representation for text. In our graph representation vertices represent words or phrases and edges represent relations between contiguous words or phrases. The graph representation also includes dependency information. Our text representation is suitable for applications involving the identification of relevance or paraphrases across texts, where word-order information would be useful. We show that this word-order based graph representation performs better than a dependency tree representation while identifying the relevance of one piece of text to another.

Parallel marking algorithms use multiple threads to walk through the object heap graph and mark each reachable object as live. Parallel marker threads mark an object "live" by atomically setting a bit in a mark-bitmap or a bit in the object header. Most of these parallel algorithms strive to improve the marking throughput by using work-stealing algorithms for load-balancing and to ensure that all participating threads are kept busy. A purely "processor-centric" load-balancing approach in conjunction with a need to atomically set the mark bit, results in significant contention during parallel marking. This limits the scalability and throughput of parallel marking algorithms.

We describe the Collie collector, a fully concurrent compacting collector that uses transactional memory techniques to achieve wait-free compaction. The collector uses compaction as the primary means of reclaiming unused memory, and performs "individual object transplantations" as transactions. We introduce new terms and requirements useful for analyzing concurrent relocating collectors, including definitions of referrer sets, object transplantation and the notion of individually transplantable objects. The Collie collector builds on these terms and on a detailed analysis of an object's legal states during compaction.

For the past several years, students in computer-science courses have been assigned work on open-source project development. The literature is replete with examples. Yet an instructor desiring to incorporate OSS into a course often has difficulty finding a suitable project and developing a fruitful interaction with its personnel. This paper reports on a survey of managers of OSS projects on how they have interacted with classes and students, and on what faculty can do to work with them effectively. Our findings indicate that instructors need to seek out projects weeks or months in advance, and need to be personally involved in OSS development themselves, and that they need to give their students a good background in design and testing. As a help for instructors looking for a project, we describe several OSS projects that have benefited from student contributions.

Classroom response devices, such as clickers, have proved effective in improving student engagement during class time. We performed a study to investigate how much of this improvement was due to heightened accountability, either because students were required to take and pass a pre-quiz over the lecture material, or because students were given credit for each answer submitted. We found that the presence of a pre-quiz was associated with a much higher response rate, 38.5% vs. 29.3%. Giving credit for answering questions also boosted the response rate, from 30.3% to 43.2%. We also found that asking more questions during class tended was associated with a lower response rate. When only one question was asked, the response rate was above 60%, but if more than five questions were asked, the response rate was barely 30%. These findings suggest that accountability is important in making effective use of classroom response devices.

Quality of a review can be identified by reviewing a review. Quantifiable factors that help identify the quality of a review include quality and tone of review comments, and the number of tokens each contains. We use machine-learning techniques such as latent semantic analysis (LSA) and cosine similarity to classify comments based on their quality and tone. Our paper details experiments that were conducted on student review and metareview data by using different data pre-processing steps. We compare these pre-processing steps and show that when applied to student review data, they help improve data quality by providing better text classification. Our technique helps predict metareview scores for student reviews.

Reviews are text-based feedback provided by reviewers to authors. The quality of a review can be determined by identifying how relevant it is to the work that the review was written for as well as its similarity to existing well-written and coherent reviews. Relevance between two pieces of text can be determined by identifying semantic and syntactic similarities between them. In this paper, we make use of string-based metrics that incorporate concepts of paraphrasing and plagiarism to determine matching between texts. We use a graph-based text representation technique. We use the k-nearest neighbor classification algorithm to build a supervised model and classify text as LOW, MEDIUM or HIGH based on values of the metrics. We evaluate our approach on three data sets from student assignments and show that our model achieves an average accuracy of 63%.

No abstract available.

Active-learning exercises are an effective use of class time to bring about desired learning outcomes. Instead of listening to a lecture, students are engaged in tasks that allow them to discover new knowledge, or apply what they have just learned. A barrier to wide usage of active-learning exercises is the need to design them, since few are available in textbooks, technical papers, or on the Web. The work reported in this paper demonstrates that students can design active-learning exercises that are worthy of being used in CS1 and CS2. This frees the instructor from having to write all the exercises him/herself. This paper makes three contributions: a methodology for creating student-generated active-learning exercises, several exercises for teaching difficult concepts in CS1 and CS2, and guidance about the kinds of active-learning exercises that students will enjoy and learn most from.

Open-source project artifacts can be used to inject realism into software engineering courses or lessons on open-source software development. However, the use of open-source projects presents challenges for both educators and for students. Educators must search for projects that meet the constraints of their classes, and often must negotiate the scope and terms of the project with project managers. For students, many available open-source projects have a steep learning curve that inhibits them from making significant contributions to the project and benefiting from a "realistic" experience. To alleviate these problems and to encourage cross-institution collaboration, we have created the Repository for Open Software Education (ROSE) and have contributed three open-source projects intended for an undergraduate computer science or software engineering course. The projects in ROSE are education-friendly in terms of a manageable size and scope, and are intended to be evolved over many semesters. All projects have a set of artifacts covering all aspects of the development process, from requirements, design, code, and test. We invite other educators to contribute to ROSE and to use projects found on ROSE in their own courses.

Wikis may be on track to take the academic world by storm. Though researchers have used them as collaborative tools for more than a decade, it is only in the past year or two that they have become widespread in education. Of the articles published by SIGCSE on wikis, nearly two-thirds (30 out of 46) of them gave appeared since the beginning of 2006. References to wikis in the educational database ERIC are approximately doubling each year. What is it about wikis that has suddenly made them so attractive? Among other things, it is the fact that collaboration becomes so easy. Students have the opportunity to revise each other’s work without the need to send documents back and forth. Because an edit history is kept, it is easy to see how much work each student has done—and to verify that it has not been downloaded from a third-party source on the eve of the due date. Wikis have been used for a wide variety of assignments, from discussion boards to writing a textbook from student contributions. This panel will present several collaborativelearning exercises that have been carried out with wikis, and give advice to instructors who want to use wikis in their classes.

No abstract available.

Course Web sites are fast becoming standard features of college courses. Some students expect all their courses to have such sites, just like they expect them to have syllabi. Course Web sites help professors communicate with students and students communicate with each other. They might summarize lectures, present assignments, serve as repositories of examples developed by students as well as the professor, and provide links to additional related information on the Web.This panel will discuss various aspects of course Web sites, focusing on their development, maintenance, and use. We will share experiences by presenting examples, highlighting successes and failures, and describing our hopes and concerns for the future. The panel will welcome comments from those in the audience with similar and even contradictory experiences. Our intention is to broaden participants' thinking on the implementation and use of course Web sites and spawn insights that might lead to more effective use of this important course component.

More companies are requiring their employees to sign acceptable-use policies for Internet computers. Some employees are unaware of the implications of the policies, and do not realize the extent to which their activities can be monitored by computer technicians. In academia, three important cases of "unacceptable use" are those of Dean Ronald F. Thiemann, Professor Eric Neil Angevine, and Superintendent Robert Herrold. All three lost, or resigned from, their positions after pornography was discovered on their employer-owned computers. Several issues regarding "acceptable use" are common to all the cases, including privacy rights, the right of the institution to control its equipment, and who might see what is stored on that equipment. This paper explores these questions, and suggests guidelines for employers and employees.

Password security is essential to the security of information systems. Human fallibility makes it nearly impossible to follow all of the recommended rules simultaneously. A user with many different passwords, frequently changing, will be forced to write them down somewhere. Some systems constrain them to have a certain minimum length, or to require them to contain a combination of letters and numbers. Some systems also impose maximum lengths, and some prohibit special characters. The lack of common standards for passwords makes it difficult for a user to remember which password is used for which system. To make matters worse, systems frequently revoke a user's access after a password has been incorrectly entered as few as three times. What is needed, then, is an analysis of passwords that takes both human factors and security into account. We must recognize that what really matters is the security of the total system-offline as well as online. This paper explores the tradeoffs that need to be made to achieve maximum security in everyday use by forgetful users.

We have implemented a peer-grading system for review of student assignments over the World-Wide Web and used it in approximately eight computer-science courses. Students prepare their assignments and submit them to our Peer Grader (PG) system. Other students are then assigned to review and grade the assignments. The system allows authors and reviewers to communicate with authors being able to update their submissions. Unique features of our approach include the ability to submit arbitrary sets of Web pages for review, and mechanisms for encouraging careful review of submissions. We have used the system to produce high-quality compilations of student work. Our assignment cycle consists of six phases, from signing up for an assignment to Web publishing of the final result. Based upon our experience with PG, we offer suggestions for improving the system to make it more easily usable by students at all levels.

Static chains and displays are two techniques used by compilers to implement variable references for block-structuredprogramming languages. Both methods impose significant run-time overhead during procedure calls and variable references. This paper investigates several optimizations to improve upon common practice. The effectiveness of the various schemes is measured by collecting dynamic reference traces from executing Ada programs and assigning costs for procedure calls and variable references using two cost models. One model assumes a complex instruction-set architecture and the other assumes a reduced instruction-set architecture. Optimizations to the static chain scheme are found to significantly reduce its overall cost and approach the cost of the display scheme while using fewer registers than a display.

Object-oriented programming languages tend to allocate and deallocate blocks of memory very frequently. The growing popularity of these languages increases the importance of high-performance memory allocation. For speed and simplicity in memory allocation, the buddy system has been the method of choice for nearly three decades. A software realization incurs the overhead of internal fragmentation and of memory traffic due to splitting and coalescing memory blocks. This paper presents a simple hardware design for buddy-system allocation that takes advantage of the speed of a pure combinational-logic implementation. Two binary trees formed by anding and oring propagate information about the allocation status of blocks and subblocks. They implement a nonbacktracking search for the address of the first free block that is large enough to satisfy a request. Although the buddy system may allocate a block that is much larger than the requested size, the logic that finds a free block can be augmented by a "bit-flipper" to relinquish the unused portion at the end of the block. This effectively eliminates internal fragmentation. Simulation results show that the buddy system modified in this way uses less memory in most, though not all, programs than the unmodified buddy. Hence, the hardware buddy-system allocator is faster and uses memory more efficiently than the standard software approach.

Object-oriented systems exhibit a very high rate of object creation, but most of the objects are short-lived. As a result, memory management overhead is significant. The paper evaluates an application-specific coprocessor architecture to speed up object creation and memory reclamation in object-oriented systems. The architecture supports a bit-vector approach to dynamic storage allocation and liberation. Newly created objects reside in a cache which is reference counted. The paper presents measurements of the performance of this coprocessor design. Simulation results show that 50% to 70% of objects die before they age out of the cache, greatly reducing the number of references to main memory. Overall, more than 60% of memory traffic is saved by the proposed scheme, and the interval between main-memory garbage collections is extended by more than 60%.>

Object-oriented systems exhibit a very high rate of object creation, but most objects are short-lived. As a result, memory-management overhead is significant. An application-specific coprocessor architecture to speed up object creation and memory reclamation in object-oriented systems is described. The architecture supports a bit-vector approach to dynamic storage allocation and liberation. Novel created objects reside in a cache that is reference counted. Most objects are expected to die before they age out of the cache, drastically reducing the number of references to main memory. Many existing computer architectures would require only minor compiler modification to incorporate and benefit from this coprocessor.>

A model for predicting multiprocessor performance on iterative algorithms is developed. Each iteration consists of some amount of access to global data and some amount of local processing. The iterations may be synchronous or asynchronous, and the processors may or may not incur waiting time, depending on the relationship between the access time and processing time. The effect on performance of the speed of the processor, memory, and the interconnection network is studied. The model also illustrates the significant impact on performance of decomposing an algorithm into parallel processes. The model's predictions are calibrated with experimental measurements.>

The Intel 432 is noteworthy as an architecture incorporating a large amount of functionality that most other systems perform by software. It has, in effect, “migrated” this functionality from the software into the microcode and hardware. The benefits of functional migration have recently been a subject of intense controversy, with critics claiming that a complex architecture is inherently less efficient than a simple architecture with good software support. This paper examines the performance impact of the incorporation of several kinds of functionality into the Intel 432. Among these are the addressing structure, the caches, instruction alignment, the buses, and the way that garbage collection is handled. A set of several benchmarks is used to quantify the performance effect of each of these decisions. The results indicate that the 432 could have been speeded up very significantly if a small number of implementation decisions had been made differently, and if incrementally better technology had been used in its construction. Even with these modifications, however, the 432 would still have only one-fourth to one times the speed of its contemporaries. These figures may represent the real cost of the 432's style of object-based programming environment.

As modular programming grows in importance, the efficiency of procedure calls assumes an ever more critical role in system performance. Meanwhile, software designers are becoming more aware of the benefits of object-oriented programming in structuring large software systems. But object-oriented programming requires a good deal of support, which can best be distributed between the compiler and architectural levels. A major part of this support relates to the execution of procedure calls. Must such support exact an unacceptable performance penalty? By considering the case of the Intel 432, a prominent object-oriented architecture, we argue that it need not. The 432 provided all the facilities needed to support object orientation. Though its procedure call was slow, the reasons were only tenuously related to object orientation. Most of the inefficiency could be removed in future designs by the adoption of a few new mechanisms: stack-based allocation of contexts, a memory-clearing coprocessor, and the use of multiple register sets to hold addressing information. These proposals offer the prospect of an object-oriented procedure call that can, on average, be performed nearly as fast as an ordinary unprotected procedure call.

Traditionally, instruction sets have included separate instructions for manipulating different data types, such as integers and real numbers.A long-discussed but seldomimplemented alternative has been tagging, where the type information is stored adjacent to the data itself, rather than being inferred from the instructions.Over the years, several advantages have been claimed for tagging, including improving program reliability, saving memory, and facilitating a high-level language architecture.For computers that are not specifically oriented toward dynamically typed languages, we show that tagging fails to achieve these advantages completely, and that most of 'them can be better achieved by extensions of other mechanisms.

article The influence of parallel decomposition strategies on the performance of multiprocessor systems Share on Authors: Dalibor Vrsalovic Department of Computer Science Carnegie-Mellon University Department of Computer Science Carnegie-Mellon UniversityView Profile , Edward F. Gehringer Department of Computer Science Carnegie-Mellon University Department of Computer Science Carnegie-Mellon UniversityView Profile , Zary Z. Segall Department of Computer Science Carnegie-Mellon University Department of Computer Science Carnegie-Mellon UniversityView Profile , Daniel P. Siewiorek Department of Computer Science Carnegie-Mellon University Department of Computer Science Carnegie-Mellon UniversityView Profile Authors Info & Claims ACM SIGARCH Computer Architecture NewsVolume 13Issue 3June 1985 pp 396–405https://doi.org/10.1145/327070.327372Online:01 June 1985Publication History 26citation241DownloadsMetricsTotal Citations26Total Downloads241Last 12 Months1Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

A capability system which supports very small objects can achieve flexible and efficient protection. This paper presents a scheme for representing both large and small entitles In a computation, down to integers and character strings, as objects. This is achieved by a generalization of tagged memory to encompass extended data types; and by the use of variable-length capabilities, which can be very short if they are close to the object they reference. As developed here, the design assumes a single, systemwide virtual-address space, and a stack architecture; but it could probably be modified for use In other environments.