This research-to-practice full paper introduces a novel graduate course, Human Errors in Software Development (HESD) based on a metacognition framework. Software defects pose a significant threat to the reliability and safety of computer systems, incurring trillions of dollars in costs globally. Addressing and rectifying defects in programs present a formidable challenge for Computer Science (CS) students. Given the critical role of cognition in software development, there is a pressing need for a course designed to train students to address human errors in various cognitive activities of software development. To our knowledge, there is currently no semester-long nor quarter-long university course offering comprehensive training to students on handling human errors in software development. HESD equips students with a profound understanding of the cognitive mechanisms underlying human errors in software development. It aims to enhance students' awareness and cognitive abilities to proactively prevent human errors in software development and, consequently, reduce defects in programs they develop. HESD comprises three stages: Stage I provides students with explicit knowledge of human errors in software development; Stage II fosters students' awareness and regulation abilities to effectively address human errors during software development; Stage III encourages students to apply acquired knowledge and skills in diverse contexts. The newly designed course was delivered to master's students at a large public university over a 14-week semester. Nine students in Software Engineering were enrolled in the course. Comprehensive surveys were used to evaluate the course's attractiveness and usefulness to students. The average satisfaction score was 4.2 (Min = 3, Max = 5, SD = 0.8) in a five-point Likert scale (where 1 means “very dissatisfied” and 5 means “very satisfied”), signifying that the students were quite satisfied with the course. A survey designed for assessing human error knowledge, awareness and regulation ability was filled out by the students before the course and at the end of the course. Results showed that the course has significantly enhanced students' Human Error Knowledge in software development by 153%, improved Human Error Awareness in software development by 62%, and elevated students' Human Error Regulation by 63%. The students found the course to be highly interesting, practical, and valuable.

Emerging Internet of Things (IoT) platforms provide sophisticated capabilities to automate IoT services by enabling occupants to create trigger-action rules. Multiple trigger-action rules can physically interact with each other via shared environment channels, such as temperature, humidity, and illumination. We refer to inter-rule interactions via shared environment channels as a physical inter-rule vulnerability. Such vulnerability can be exploited by attackers to launch attacks against IoT systems. We propose a new framework to proactively discover possible physical inter-rule interactions from user requirement specifications (i.e., descriptions) using a deep learning approach. Specifically, we utilize the Transformer model to generate trigger-action rules from their associated descriptions. We discover two types of physical inter-rule vulnerabilities and determine associated environment channels using natural language processing (NLP) tools. Given the extracted trigger-action rules and associated environment channels, an approach is proposed to identify hidden physical inter-rule vulnerabilities among them. Our experiment on 27983 IFTTT style rules shows that the Transformer can successfully extract trigger-action rules from descriptions with 95.22% accuracy. We also validate the effectiveness of our approach on 60 SmartThings official IoT apps and discover 99 possible physical inter-rule vulnerabilities.

Modern code reviews tend to take a lightweight process, in which the accuracy and efficiency of identifying defects rely heavily on code reviewers’ experience. The human errors of developers, as a significant cause of software defects, is a key to identifying defects. However, there is a lack of understanding of the human error mechanisms underlying defects in code. This paper proposes an innovative code review method for identifying defects by pinpointing the scenarios that developers tend to commit errors. The method was validated by two experimental studies that involved 40 participants of about 5 years’ programming experience and modest code review experience. The experiment shows that the proposed method has significantly improved True Positives and Sensitivity by about 400%, improved Precision by approximately 200%, and reduced around one-third of False Positives. The effects were consistent across different tasks and different code reviewers.

This Research to Practice Full Paper presents a framework designed to train students in identifying human error-prone scenarios in source code. Human errors are a leading cause of defective programs written by students, and learning how to identify error-prone contexts in code can help students detect defects in their own code (debugging), identify defects in other people's code (code review), and prevent defects in similar situations in the future (defect prevention), thereby improving their skills of producing reliable programs. Despite the overwhelming diversity of defects in different programming contexts, recent scientific research has confirmed that they share limited forms of cognitive mechanisms. To identify diverse defects using a limited forms of human errors, a framework to bridge the limited cognitive forms (Human Error Modes) to diverse specific contexts (defects) is essential. The framework is known as Error-Prone Scenario Analysis, which is a method recently proposed and used for forecasting software defects that may be later introduced in code based on analysis of requirements. The purpose of this paper is to explore the extent to which the human Error-Prone Scenarios framework can be used to enhance students' abilities to identify defects in source code. The author incorporated the training framework into a graduate course called “Software Verification and Validation” as two lectures, following the ethical and approval guidelines of the University of Coimbra. The training process and materials used are reported in detail in this paper. Forty graduate students in Computer Science, spread across two classes, received this training. The results show that the training improved their performances in identifying error-prone scenarios by approximately 400%. The study demonstrates that the Error-Prone Scenario framework is effective in helping students recognize the underlying patterns of software defects that appear diverse in different programming contexts, thus enabling them to identify more defects accurately. The students who received the training highly regard the EPS training as interesting, important, and useful. The key successful experience gained from this study is using a pool of Human Error Modes, the Error-Prone Scenario for each Human Error Mode, and a graphic notation to display the mechanisms of how a Human Error Mode manifests as a defect. The research provides valuable insights that will benefit learners, educators, and professionals involved in programming, code review, and software quality assurance.

Modern code reviews tend to take a lightweight process, in which the accuracy and efficiency of identifying defects rely heavily on code reviewers’ experience. The human errors of developers, as a significant cause of software defects, is a key to identifying defects. However, there is a lack of understanding of the human error mechanisms underlying defects in code. This paper proposes an innovative code review method for identifying defects by pinpointing the scenarios that developers tend to commit errors. The method was validated by a comprehensive experimental study that involved 49 code reviewers organized in two independent groups, i.e. experimental group vs. controlled group for each other. Forty reviewers have completed the whole experiment and provided the data for statistical analysis on the effects of the approach. The experiment shows that the proposed method has significantly improved True Positives and Sensitivity by about 400%, improved Precision by approximately 200%, and reduced around one-third of False Positives. The effects were consistent across different tasks and different code reviewers.

Abstract Residual defects at the time of delivery are an important concern for safety critical software systems. Suppliers and customers are urged to get evidence for what they can do to reduce residual defects. Thus, it is meaningful to learn from historical data concerning the kinds of defects that have escaped from the existing quality assurance approaches and the factors that lead to the residual defects. A total of 3747 defects from 70 software systems developed by 29 Chinese aviation organizations were collected from acceptance tests during the last 5 years. For all these organizations, 38 domain experts from the industry assessed the process consistency to the standard built in the framework of Capability Maturity Model (CMM). Results demonstrate that the process improvement in the range of high consistency is effective in reducing total defects, as well as the minor and severe defects. The high consistency adoption of the practices in CMM Level 1 to Level 3 is more effective in reducing minor defects than severe defects. Causal analysis was performed to investigate the underlying mechanisms. Results reveal that individual cognitive failures cause 87% of severe defects. More approaches to help software developers manage their interior cognitive process are needed for improving software quality in the future. Copyright © 2015 John Wiley & Sons, Ltd.