The goal of any automatic process is to run without human intervention. However, it is not uncommon that unexpected issues recur, requiring repetitive human intervention in order to processes execute successfully. Such situations indicate that those processes were not designed with the necessary quality requirements to achieve the automation goal, and quality needs to be improved. This paper describes a case study of the quality improvements of several automatic ETL processes that required a growing amount of effort to ensure its successful execution. Key to the improvement process was the use of the ISO/IEC 25010 Standard for Systems and Software Quality Requirements and Evaluation to categorize all issues found. This approach proved beneficial in two important ways: first, it created a clear understanding of the overall ETL processes quality problems, second, it made obvious which issues deserved uttermost attention.

The categorization of source code artifacts affects how the overall product is measured and consequently how these measurements are interpreted. When measuring complexity, for instance, failing to distinguish test and generated code will affect complexity measurements possibly leading to an erroneous interpretation of the overall product complexity. Although categorization problems are known, there seems to be little attention given to this subject in the literature. In this paper, we introduce a categorization for source code artifacts and present an empirical study providing evidence of each category. Artifacts are divided into production and test code, and then these categories are sub-divided into manually-maintained, generated, library, and example code. By analyzing 80 Java and C# industrial systems, we have found evidence of the majority of categories. We show that in average production code only accounts for 60% of a product volume. Also, we have found that for some systems the overall percentage of test and generated code, each can account to over 70% and of library code to over 40%. Finally we discuss the difficulties of distinguishing source code artifacts and conclude with directions for further research.

Software metrics have been proposed as instruments, not only to guide individual developers in their coding tasks, but also to obtain high-level quality indicators for entire software systems. Such system-level indicators are intended to enable meaningful comparisons among systems or to serve as triggers for a deeper analysis.Common methods for aggregation range from simple mathematical operations (e.g. addition and central tendency) to more complex methodologies such as distribution fitting, wealth inequality metrics (e.g. Gini coefficient and Theil Index) and custom formulae.However, these methodologies provide little guidance for interpreting the aggregated results or to trace back to individual measurements.To resolve such limitations, a two-stage rating approach has been proposed where (i) measurement values are compared to thresholds to summarize them into risk profiles, and (ii) risk profiles are mapped to ratings.In this paper, we extend our approach for deriving metric thresholds from benchmark data into a methodology for benchmark-based calibration of two-stage aggregation of metrics into ratings.We explain the core algorithm of the methodology and we demonstrate its application to various metrics of the SIG quality model, using a benchmark of 100 software systems.We present an evaluation of the sensitivity of the algorithm to the underlying data.

When analyzing software systems we face the challenge of how to implement a particular analysis for different programming languages. A solution for this problem is to write a single analysis using a code query language, abstracting from the specificities of languages being analyzed. Over the past ten years many code query technologies have been developed, based on different formalisms. Each technology comes with its own query language and set of features. To determine the state of the art of code querying we compare the languages and tools for seven code query technologies: Grok, Rscript, JRelCal, Semmle Code, JGraLab, CrocoPat and JTransformer. The specification of a package stability metric is used as a running example to compare the languages. The comparison involves twelve criteria, some of which are concerned with properties of the query language (paradigm, types, parametrization, polymorphism, modularity, and libraries), and some of which are concerned with the tool itself (output formats, interactive interface, API support, interchange formats, extraction support, and licensing). We contextualize the criteria in two usage scenarios: interactive and tool integration. We conclude that there is no particularly weak or dominant tool. As important improvement points, we identify the lack of library mechanisms, interchange formats, and possibilities for integration with source code extractors.

The software life-cycle of applications supporting space missions follows a rigorous process in order to ensure the application compliance with all the specified requirements. Ensuring the correct behavior of the application is critical since an error can lead, ultimately, to the loss of a complete space mission. However, it is not only important to ensure the correct behavior of the application but also to achieve good product quality since the applications need to be maintained for several years. Then, the question arises, is a rigorous process enough to guarantee good product maintainability? In this paper we assess the software product maintainability of two simulators used to support space missions. The assessment is done using both a standardized analysis, using the SIG quality model for maintainability, and a customized copyright license analysis. The assessment results revealed several quality problems leading to three lessons. First, rigorous process requirements by themselves do not ensure product quality. Second, quality models can be used not only to pinpoint code problems but also to reveal team issues. Finally, tailored analyses, complementing quality models, are necessary for in-depth investigation of quality.

A wide variety of software metrics have been proposed and a broad range of tools is available to measure them. However, the effective use of software metrics is hindered by the lack of meaningful thresholds. Thresholds have been proposed for a few metrics only, mostly based on expert opinion and a small number of observations. Previously proposed methodologies for systematically deriving metric thresholds have made unjustified assumptions about the statistical properties of source code metrics. As a result, the general applicability of the derived thresholds is jeopardized. We designed a method that determines metric thresholds empirically from measurement data. The measurement data for different software systems are pooled and aggregated after which thresholds are selected that (i) bring out the metric's variability between systems and (ii) help focus on a reasonable percentage of the source code volume. Our method respects the distributions and scales of source code metrics, and it is resilient against outliers in metric values or system size. We applied our method to a benchmark of 100 object-oriented software systems, both proprietary and open-source, to derive thresholds for metrics included in the SIG maintainability model.

Test coverage is an important indicator for unit test quality. Tools such as Clover compute coverage by first instrumenting the code with logging functionality, and then logging which parts are executed during unit test runs. Since computation of test coverage is a dynamic analysis, it presupposes a working installation of the software. In the context of software quality assessment by an independent third party, a working installation is often not available. The evaluator may not have access to the required libraries or hardware platform. The installation procedure may not be automated or documented. In this paper, we propose a technique for estimating test coverage at method level through static analysis only. The technique uses slicing of static call graphs to estimate the dynamic test coverage. We explain the technique and its implementation. We validate the results of the static estimation by statistical comparison to values obtained through dynamic analysis using Clover. We found high correlation between static coverage estimation and real coverage at system level but closer analysis on package and class level reveals opportunities for further improvement.