An overwhelming number of users access app repositories like App Store/Google Play and social media platforms like Twitter, where they provide feedback on digital experiences. This vast textual corpus comprising user feedback has the potential to unearth detailed insights regarding the users’ opinions on products and services. Various tools have been proposed that employ natural language processing (NLP) and traditional machine learning (ML) based models as an inexpensive mechanism to identify requirements in user feedback. However, they fall short on their classification accuracy over unseen data due to factors like the cost of generating voluminous de-biased labeled datasets and general inefficiency. Recently, Van Vliet et al. [1] achieved state-of-the-art results extracting and classifying requirements from user reviews through traditional crowdsourcing. Based on their reference classification tasks and outcomes, we successfully developed and validated a deep-learning-backed artificial intelligence pipeline to achieve a state-of-the-art averaged classification accuracy of ∼87% on standard tasks for user feedback analysis. This approach, which comprises a BERT-based sequence classifier, proved effective even in extremely low-volume dataset environments. Additionally, our approach drastically reduces the time and costs of evaluation, and improves on the accuracy measures achieved using traditional ML-/NLP-based techniques.

Rapid advancements in synthetic biology and nucleic acid synthesis, in particular concerns about its intentional or accidental misuse, call for more sophisticated screening tools to identify genes of interest within short sequence fragments. One major gap in predicting genes of concern is the inadequacy of current tools and ontologies to describe the specific biological processes of pathogenic proteins. The objective of this work is to design software that sensitively assigns taxonomic classifications, functional annotations, and biological processes of interest to short nucleotide sequences of unknown origin (50bp-1,000bp). The overarching goal is to perform sensitive characterization of short sequences and highlight specific pathogenic biological processes of interest (BPoIs). The SeqScreen software executes these tasks in analytical workflows with Nextflow and outputs results in a tab-delimited report. Local and global alignments differentiate hits to taxonomically-related sequences from similar but unrelated sequences, and an ensemble approach leverages multiple tools and databases to assign a variety of functional terms to each query sequence. Final biological process assessments are made from the predicted functional annotations, which leverage information in pre-existing databases, as well as new custom biocurations. Machine learning models predict each biological process of interest on large protein databases before incorporation into the SeqScreen framework to streamline computational efficiency, ensure reproducible results, allow for version control, and facilitate the review of the automated predictions by expert biocurators. The SeqScreen source code is available at https://gitlab.com/treangenlab/seqscreen.

Security attacks on medical devices have been shown to have potential safety concerns. Because of this, stakeholders (device makers, regulators, users, etc.) have increasing interest in enhancing security in medical devices. An effective means to approach this objective is to integrate systematic security requirements elicitation and analysis into the design and evaluation of medical device software. This paper extends the sequence-based enumeration approach, a systematic approach for defining the behavior of embedded software, to analyze the requirement documents of a medical device for the purpose of eliciting security requirements. As a proof of concept, we apply our approach on a concrete case study, which shows that the extended approach is useful for identifying sequences of medical device events that might be harmful to the patient, for example because the events are initiated by an active adversary trying to use the device in a malicious way. We then show how security requirements may be formulated based on the identified threats. By exploring these sequences systematically, the developers can reliably assess what, where, and how the security threats may manifest in their system, what the safety implications are, and finally they can evaluate the resulting requirements and mitigations.

Many systems make use of concurrent tasks, however it is often difficult to test concurrent design. Therefore, many test cases are simplified and do not fully test all concurrency aspects of the system. We encountered this problem when analyzing test cases for concurrent flight software at NASA. To address this problem, we developed and evaluated a model based testing (MBT) technique for testing of concurrent systems. Using MBT, the tester creates a model, which is based on the requirements of the system under test (SUT), and lets the computer generate innumerable test cases automatically from the model. We evaluate the effectiveness of the technique using Microsoft's Spec Explorer MBT tool. We apply the technique on NASA's Core Flight Software (cFS) software bus module API, which is based on a concurrent publisher-subscriber architecture style and is a safety-critical system. We describe how we created a test automation architecture for testing concurrent inter-task communication as carried out by the software bus. We also investigate the type of issues the technique for testing of concurrent systems can find as well as what degree of code coverage it can achieve.

Testing is necessary for all types of systems, but becomes difficult when the tester cannot easily determine whether the system delivers the correct result or not. NASA's Data Access Toolkit allows NASA analysts to query a large database of telemetry data. Since the user is unfamiliar with the data and several data transformations can occur, it is impossible to determine whether the system behaves correctly or not in full scale production situations. Small scale testing was already conducted manually by other teams and unit testing was conducted on individual functions. However, there was still a need for full scale acceptance testing on a broad scale. We describe how we addressed this testing problem by applying the idea of metamorphic testing [1]. Specifically, we base it on equivalence of queries and by using the system itself for testing. The approach is implemented using a model-based testing approach in combination with a test data generation and test case outcome analysis strategy. We also discuss some of the issues that were detected using this approach.

Model-based Testing (MBT), where a model of the system under test's (SUT) behavior is used to automatically generate executable test cases, is a promising and versatile testing technology. Nevertheless, adoption of MBT technologies in industry is slow and many testing tasks are performed via manually created executable test cases (i.e. test programs such as JUnit). In order to adopt MBT, testers must learn how to construct models and use these models to generate test cases, which might be a hurdle. An interesting observation in our previous work is that the existing manually created test cases often provided invaluable insights for the manual creation of the testing models of the system. In this paper we present an approach that allows the tester to first create and debug a set of test cases. When the tester is happy with the test cases, the next step is to automatically generate a model from the test cases. The generated model is derived from the test cases, which are actions that the system can perform (e.g. a button clicks) and their expected outputs in form of assert statements (e.g. assert data entered). The model is a Finite State Machine (FSM) model that can be employed with little or no manual changes to generate additional test cases for the SUT. We successfully applied the approach in a feasibility study to the NASA Data Access Toolkit (DaT), which is a web-based GUI. One compelling finding is that the test cases that were generated from the automatically generated models were able to detect issues that were not detected by the original set of manually created test cases. We present the findings from the case study and discuss best practices for incorporating model generation techniques into an existing testing process.

This article introduces the Architecture Discovery and Analysis Method (ADAM). ADAM supports the discovery of module and runtime views as well as the analysis of quality attributes, such as testability, performance, and maintainability, of software systems. The premise of ADAM is that the implementation constructs, architecture constructs, concerns, and quality attributes are all influenced by the external entities (e.g., libraries, frameworks, COTS software) used by the system under analysis. The analysis uses such external dependencies to identify, classify, and review a minimal set of key source-code files supported by a knowledge base of the external entities. Given the benefits of analyzing external dependencies as a way to discover architectures and potential risks, it is demonstrated that dependencies to external entities are useful not only for architecture discovery but also for analysis of quality attributes. ADAM is evaluated using the NASA's Space Network Access System (SNAS). The results show that this method offers systematic guidelines for discovering the architecture and locating potential risks (e.g., low testability and decreased performance) that are hidden deep inside the system implementation. Some generally applicable lessons for developers and analysts, as well as threats to validity are also discussed.

Exchangeability between software components such as operating systems, middleware, databases, and hardware components is a common requirement in many software systems. One way to enable exchangeability is to promote indirect use through a common interface and an implementation for each component that wraps the original component. As developers use the interface instead of the underlying component, they assume that the software system will behave in a specific way independently of the actual component in use. However, differences in the implementations of the wrappers may lead to different behavior when one component is changed for another, which might lead to failures in the field. This work reports on a simple, yet effective approach to detect these differences. The approach is based on tool-supported reviews leveraging lightweight static analysis and machine learning. The approach is evaluated in a case study that analyzes NASA's Operating System Abstraction Layer (OSAL), which is used in various space missions. We detected 84 corner-case issues of which 57 turned out to be bugs that could have resulted in runtime failures.

We present a case study that evaluates the applicability and effectiveness of model-based testing in detecting bugs in real-world, mission-critical systems. NASA's Operating System ion Layer (OSAL) is the subject system of this paper. The OSAL is a reusable framework that wraps several operating systems (OS) and is used extensively in NASA's flight software missions. We developed a suite of behavioral models, represented as hierarchical finite state machines (FSMs), of the core file system API and generated a large number of test cases automatically. We then automatically executed these test cases against the OSAL. The results show that the OSAL is a high quality product. Naturally, due to the systematic and rigorous nature of MBT, we detected a few previously unknown “corner-case” bugs and issues, which escaped traditional manual testing and code reviews. We discuss the MBT architecture, the detected bugs, the code coverage of generated tests, as well as threats to validity of the study.

New research is underway at the FDA to investigate the benefits of integrating architecture analysis into safety evaluations of medical-device software. Due to the complexity in setting up testing environments for such software, the FDA is unable to conduct large-scale safety testing, instead, it must rely on other techniques to build an argument for whether the software is safe or not. The architecture analysis approach, formalized using relational algebra, is based on reconstructing abstract, yet precise, architectural views from source code to help build such arguments about safety. This paper discusses the use of the formal approach to analyze the Computer-Assisted Resuscitation Algorithm (CARA) software, which controls an infusion pump designed to provide automated assistance for transfusing blood. The results suggest that a) architecture analysis offers many insights related to software quality in general and testability (i.e., the ease of testing) and its impact on safety in particular, and b) architectural analysis results can be used to help configure static analysis tools to improve their performance for verifying safety properties.

Modern software systems are difficult to test due to their distributed nature, and increased security complicates testing even further. Our hypothesis is that some security vulnerabilities are actually introduced due to developerspsila need to facilitate testing that software requirements have been implemented correctly. If these temporary security vulnerabilities are not removed before the software is delivered, there is a great risk that they may become fielded security vulnerabilities.In this paper, we study the relationship between such security vulnerabilities and developers' need to improve the testability of an application to facilitate unit and integration testing. We trace detected vulnerabilities to characteristics of the software that made testing difficult and therefore led to testability improvements. We discuss how the need to increase testability may relate to a form of developer usability, and what the ways of dealing with the problem of security vulnerabilities as a consequence of increasing testability are.

Software architecture has become essential in the struggle to manage today's increasingly large and complex systems. Software architecture views are created to capture important system characteristics on an abstract and, thus, comprehensible level. As the system is implemented and later maintained, it often deviates from the original design specification. Such deviations can have implication for the quality of the system, such as reliability, security, and maintainability. Software architecture compliance checking approaches, such as the reflexion model technique, have been proposed to address this issue by comparing the implementation to a model of the systems' architecture design. However, architecture compliance checking approaches focus solely on structural characteristics and ignore behavioral conformance. This is especially an issue in Systems-of-Systems. Systems-of-Systems (SoS) are decompositions of large systems, into smaller systems for the sake of flexibility. Deviations of the implementation to its behavioral design often reduce the reliability of the entire SoS. An approach is needed that supports the reasoning about behavioral conformance on architecture level.In order to address this issue, we have developed an approach for comparing the implementation of a SoS to an architecture model of its behavioral design. The approach follows the idea of reflexion models and adopts it to support the compliance checking of behaviors. In this paper, we focus on sequencing properties as they play an important role in many SoS. Sequencing deviations potentially have a severe impact on the SoS' correctness and qualities. The desired behavioral specification is defined in UML sequence diagram notation and behaviors are extracted from the SoS implementation. The behaviors are then mapped to the model of the desired behavior and the two are compared. Finally, a reflexion model is constructed that shows the deviations between behavioral design and implementation. This paper discusses the approach and shows how it can be applied to investigate reliability issues in SoS.

Systems often collaborate to form a system-of-systems (SoS) and together fulfill some larger task. Correctness and performance issues in the interaction between participating systems are frequent occurrences and decrease the reliability of the entire SoS. We are currently developing an analysis framework to automatically compare a model of the desired interaction behavior (specification) to a model that is retrieved from the system execution and detect deviations between the two. The specification, the observed interaction behavior, and the evaluation result are presented in behavioral diagrams to be analyzed by the user.

Recent experience and feedback from panels indicates that what the audience likes best is the chance to ask questions, particularly regarding things that might solve problems on their development projects or in their research studies. This panel has been held at every ISSRE since 1997, and has been highly popular. Originally the panel was chaired by John Musa, but Mike Hinchey took over that role last year. The panel has no presentations, only questions. The questions can be on anything, ranging from theory to details of application. The panelists have been selected to make available wide and extensive experience in the field.

Inspections are widely used and studies have found them to be effective in uncovering defects. However, there is less data available regarding the impact of inspections on different defect types and almost no data quantifying the link between inspections and desired end product qualities. This paper addresses this issue by investigating whether design inspection checklists can be tailored so as to effectively target certain defect types without impairing the overall defect detection rate. The results show that the design inspection approach used here does uncover useful design quality issues and that the checklists can be effectively tailored for some types of defects.

GQM+Strategies is a measurement approach that builds on the well-tested GQM approach to planning and implementing software measurement. Although GQM has proven itself useful in a variety of industrial settings, one recognized weakness is the difficulty for GQM users to link software measurement goals to higher-level goals of the organization in which the software is being developed. This linkage is important, as it helps to justify software measurement efforts and allows measurement data to contribute to higher-level decisions. GQM+strategies provides mechanisms for explicitly linking software measurement goals, to higher-level goals for the software organization, and further to goals and strategies at the level of the entire business.

During its lifecycle, software experiences numerous changes that are either due to bug fixes or to the incorporation of new features. Implementing such changes is often more difficult than expected, which frequently leads to underestimation of the associated implementation effort. This paper describes work that aims to understand change impact based on the visualization of the characteristics of change requests using standard diagrams (e.g. UML diagrams). Each diagram depicts different aspects of a software system and can illustrate factors that might drive the implementation effort. Using examples, in which we analyze three change requests, we describe how we determined the characteristics of the change requests. We furthermore discuss how the impact of change requests on diagrams relates to the implementation effort. Our overall goal is to develop a new impact analysis approach that helps determine software change impact based on the use of specific diagrams

We present an experimental study which demonstrates that model checking techniques can be effective in finding synchronization errors in safety critical software when they are combined with a design for verification approach. We apply the concurrency controller design pattern to the implementation of the synchronization operations in Java programs. This pattern enables a modular verification strategy by decoupling the behaviors of the concurrency controllers from the behaviors of the threads that use them using interfaces specified as finite state machines. The behavior of a concurrency controller can be verified with respect to arbitrary numbers of threads using infinite state model checking techniques, and the threads which use the controller classes can be checked for interface violations using finite state model checking techniques. We present techniques for thread isolation which enables us to analyze each thread in the program separately during interface verification. We conducted an experimental study investigating the effectiveness of the presented design for verification approach on safety critical air traffic control software. In this study, we first reengineered the Tactical Separation Assisted Flight Environment (TSAFE) software using the concurrency controller design pattern. Then, using fault seeding, we created 40 faulty versions of TSAFE and used both infinite and finite state verification techniques for finding the seeded faults. The experimental study demonstrated the effectiveness of the presented modular verification approach and resulted in a classification of faults that can be found using the presented approach.

At NASA, more and more functionality is implemented in software. Change requests tend to arrive late, making software flexibility an important aspect. However, such changes present a high risk and are more likely to introduce defects and to violate flexibility properties of the software, causing its "degeneration." In this paper, we present some preliminary results from a recent study of Software Architecture Flexibility. The study was conducted as an experiment with students using the TSAFE experimental testbed. The main conclusion is that constructs for flexibility are not automatically understood, used, and followed by new developers. Better ways to describe the intention with certain constructs in combination with assigning responsibility for making sure that the implementation adheres to the intended architecture and its design rationale could provide a successful approach for avoiding degeneration of built-in flexibility.

The software architecture is one of the most crucial artifacts within the lifecycle of a software system. Decisions made at the architectural level directly enable, facilitate, hamper, or interfere with the achievement of business goals as well as meeting functional and quality requirements. Software architectures are also essential for the success of product line engineering. In this work, we summarize how, from our practical experience, static architecture evaluation contributes to architecture development. We also describe the different purposes of architectural evaluations.

Planning and management of software sustainment is impaired by a lack of consistently applied, practical measures. Without these measures, it is impossible to determine the effect of efforts to improve sustainment practices. In this paper we provide a context for evaluating sustainability and discuss a set of measures developed at the Software Engineering Institute at Carnegie Mellon University.

Software systems evolve over time and undergo changes that can lead to a degeneration of the systems' architecture. Degeneration may eventually reach a level where a complete redesign of the software system is necessary, which is a task that requires significant effort. In this paper, we start by presenting examples of such degeneration and continue with an analysis of technologies that can be used to diagnose degeneration. These technologies can be employed in identifying, degeneration so that it can be treated as early as possible, before it is too late and the system has to undergo a costly redesign.

Abstract Welcome to this special issue of the Journal of Software Maintenance and Evolution on process diversity. We, the guest editors, are very pleased to be able to include three fine papers on this subject, and we hope that our readers will find them stimulating and informative. Copyright © 2003 John Wiley & Sons, Ltd.

Selecting technologies for developing software is a crucial activity in software projects. Defect reduction is an example of an area in which software developers have to decide what technologies to use. CeBASE is a NSF funded project that has the role of improving software development by providing decision support on the selection of techniques and tools. The decision support is based on empirical data organized in experience bases and refined into high-level models. Empirical data is collected through various activities, for example through eWorkshops in which experts discuss important issues, and formalized using the lightweight knowledge dust to knowledge pearl process.

Constant changes cause software architectures to degenerate. In organizations where developers are involved in the maintenance and evolution, system degeneration is even more likely to occur. We have experimented with a process for detecting deviations from the intended design. The process is a quick and inexpensive process that helps to keep the system architecture from veering off course from the planned design. It is expected to improve maintainability over time. The paper describes our evaluation process, the metrics used and presents results of applying the process in a case study. The case study illustrates the difficulty involved in detecting deviations without a systematic approach. The process used to evaluate the architecture is general and straightforward. We believe the process can be reused in other contexts.

The task of finding an optimal selection of requirements for the next release of a software system is difficult as requirements may depend on each other in complex ways. The paper presents the results from an in-depth study of the interdependencies within 5 distinct sets of requirements, each including 20 high-priority requirements of 5 distinct products from 5 different companies. The results show that: (1) roughly 20% of the requirements are responsible for 75% of the interdependencies; (2) only a few requirements are singular; (3) customer-specific bespoke development tend to include more functionality- related dependencies whereas market-driven product development have an emphasis on value-related dependencies. Several strategies for reducing the effort needed for identifying and managing interdependencies are outlined. A technique for visualization of interdependencies with the aim of supporting release planning is also discussed. The complexity of requirements interdependency analysis is studied in relation to metrics of requirements coupling. Finally, a number of issues for further research are identified.

This empirical study analyzes changes in C++ source code which occurred between two releases of an industrial software product and compares them with entities and relations available in object-oriented modeling techniques. The comparison offers increased understanding of what changes can and cannot be described using such object models. The goals were to investigate if the object model in this particular project is either abstract and stable or detailed and sensitive to change, and whether or not changes made to the C++ source code are visible in the object model. Four metrics for characterization of change are formally defined and used, namely correctness, completeness, compliance, and visibility factor. The major finding is that even though many of the classes are changed, the majority of these changes turn out to be invisible in the object model. That is, changes made on the source code level are of a finer granularity than available in common object modeling concepts. This may explain why object models seem to be of little use in release-oriented development.

Impact analysis is an essential activity as cost and effort estimation is based on its outcome. Impact analysis, as performed in an industrial object-oriented system, has been analysed and evaluated in this case study. Preceding the design phase of the fourth release of the PMR system (PMR R4), the consequences of a set of new requirements were analysed. Traceability links were established between each new requirement and the objects predicted to be changed in the design object model, which represent the C++ classes in the system. The impact analysis as performed in this study was successful in the sense that the software engineers in the study prefer this way of working. When asked, they claim they are sure the number of objects to be changed is correct, while they are less sure about the number of man-hours required. The analysis of the impact analysis shows that individual objects are often predicted almost correctly, but that the number of actually changed objects is often greater than the number of predicted objects. The study also shows that tracing by interviewing knowledgeable software engineers is far more common than consulting object models and other kinds of documentation. A factor that prohibits software engineers using object models more extensively is lack of detailed information. We see a promising potential for using traceability, together with the use of existing object models, to achieve a more structured impact analysis process and accurate prediction. © 1998 John Wiley & Sons, Ltd.

Traceability defined as the ability to trace dependent items within a model and the ability to trace correspondent items in other models is advocated as a desirable property of a software development process. Potential benefits of good traceability are clearer documentation, more focussed development, increased ease of system understanding, and more precise impact analysis of proposed changes. An industry-scale project applying the analysis and design method Objectory has been examined and documented with a number of traceability examples generated from the perspective of a maintainer attempting to understand the system. Four representative examples and a categorization of traceability are presented in this study in order to provide a concrete empirical basis for the application of traceability to systems development.