A software supply chain consists of anything needed to develop and deliver a software project, including (third-party) components. Software Composition Analysis (SCA) allows for managing the security of software supply chains by identifying such components and their (security) vulnerabilities. The main goal of the empirical study presented in this paper is to investigate the effects of adopting/using over time an SCA tool like OWASP Dependency-Check (OWASP DC) in the context of the security of the software supply chain. To this end, following a cohort design, we analyzed the vulnerabilities affecting the components of the open-source (OS) Java Maven projects owned by the Apache Software Foundation (ASF) and publicly hosted on GitHub. These projects could adopt (or not) OWASP DC. The results indicate that the adoption of OWASP DC appears to be causing a significant reduction in the overall number/score of vulnerabilities, including those with a high Common Vulnerability Scoring System (CVSS) severity level. The use of OWASP DC also increased the vulnerabilities with a low severity level. Our results seem to encourage practitioners to adopt SCA to improve the security of their software supply chains.

Abstract [Context] Microservices enable the decomposition of applications into small, independent, and connected services. The independence between services could positively affect a project’s velocity, which is considered an important maintenance metric measuring the time taken to implement features and fix bugs. However, no studies have investigated the causal relationship between microservices and velocity. [Objective and Method] The goal of this study is to investigate the effect of microservices on velocity which is a common maintenance metric. The study compares projects on GitHub developed with microservices style from the beginning and similar projects using monolithic architectures. The study was conducted as a retrospective cohort study, which is a study type used to assess causality. [Results] The results did not find statistically significant differences in mean velocities in microservice-based and monolithic projects. Furthermore, the statistical adjustment performed to quantify the statistical impact of the use of microservices on velocity considering additional confounders did not find statistically significant impact from these. [Conclusions] The results did not indicate a difference between microservices-based projects and monolithic projects in terms of velocity. In addition, this study will contribute to the body of knowledge of empirical methods and be among the first works to adopt the methodology of the cohort study.

Context Replication plays a critical role in building cumulative scientific knowledge. However, in the context of Empirical Software Engineering (ESE), replication efforts face persistent difficulties, both technical and methodological, which hinder reproducibility and generalizability. We explored whether the problems were due to formal issues, such as under‐specification or miscommunication, or intrinsic reasons, i.e., the existing replication procedures may not meet the researchers’ needs. Objective To understand how ESE researchers conduct experiments in real‐world settings. Method We conducted an ethnographic study with an experimental software engineering group, using interviews, observations, and analysis of internal documentation. Results We have created conceptual and process models representing experimentation, replication, and synthesis in the target research group. These models align with mainstream procedures at a high level but often break down in practice during specialized tasks such as coordination or documentation. The experimental process observed in the group differs from common assumptions and textbook descriptions in terms of (1) the number and diversity of activities involved, (2) the presence of differentiated roles, (3) the granularity of conceptual elements, and (4) the varying perspectives across subareas or families of experiments. Conclusions The discrepancies between actual laboratory practices and idealized process models may hinder knowledge transfer and complicate replication. We plan to extend this research by involving additional research groups, for instance, through surveys, to examine the generalizability of our findings.

Context:In software engineering (SE) experiments, the way in which a treatment is applied could affect results. Different interpretations of how to apply the treatment and decisions on treatment adherence could lead to different results when data are analysed.Objective:This paper aims to study whether treatment adherence has an impact on the results of an SE experiment.Method:The experiment used as test case for our research uses Test-Driven Development (TDD) and Incremental Test-Last Development, (ITLD) as treatments. We reported elsewhere the design and results of such an experiment where 24 participants were recruited from industry. Here, we compare experiment results depending on the use of data from adherent participants or data from all the participants irrespective of their adherence to treatments.Results:Only 40% of the participants adhere to both TDD protocol and to the ITLD protocol; 27% never followed TDD; 20% used TDD even in the control group; 13% are defiers (used TDD in ITLD session but not in TDD session). Considering that both TDD and ITLD are less complex than other SE methods, we can hypothesize that more complex SE techniques could get even lower adherence to the treatment.Conclusion:Both TDD and ITLD are applied differently across participants. Training participants could not be enough to ensure a medium to large adherence of experiment participants. Adherence to treatments impacts results and should not be taken for granted in SE experiments.

Abstract Collecting and analyzing data about developers working on their development tasks can help improve development practices, finally increasing the productivity of teams. Indeed, monitoring and analysis tools have already been used to collect data from productivity tools. Monitoring inevitably consumes resources and, depending on their extensiveness, may significantly slow down software systems, interfering with developers’ activity. There is thus a challenging trade-off between monitoring and validating applications in their operational environment and preventing the degradation of the user experience. The lack of studies about when developers perceive an overhead introduced in an application makes it extremely difficult to fine-tune techniques working in the field. In this paper, we address this challenge by presenting an empirical study that quantifies how developers perceive overhead. The study consists of three replications of an experiment that involved 99 computer science students in total, followed by a small-scale experimental assessment of the key findings with 12 professional developers. Results show that non-negligible overhead can be introduced for a short period into applications without developers perceiving it and that the sequence in which complex operations are executed influences the perception of the system response time. This information can be exploited to design better monitoring techniques.

Purpose: Mindfulness is a meditation technique whose main goal involves maintaining a calm mind and training attention by focusing only on a single thing (the support) at a time; this support is usually the practitioner's breathing. The practice of mindfulness aims to improve concentration and attention, which has proven useful in knowledge-intensive and stressful work environments like technological companies. This article aims to find empirical evidence on the positive effect of the practice of mindfulness on a sample of 56 helpdesk employees working for a consulting and information technology company (Accenture) with respect to: i) their attention awareness; ii) a set of key performance indicators (KPIs); and iii) the perceived benefits of mindfulness. Method: Of the 56 recruited employees, 29 worked as managers, and 27 worked as agents answering phone calls to solve software issues of the main information system of the Andalusian Health Service, a public organization with more than 115,000 employees. Mindfulness (the treatment) was applied to 26 subjects, while the other 30 subjects were the control group. For all subjects, their attention awareness was measured using the MAAS scale. Results: Both helpdesk managers and agents significantly improved their attention awareness with respect to the control group. Regarding organizational KPIs, in general, no evidence of significant differences between groups was detected, apart from the fact that the number of phone calls answered was significantly lower in the mindfulness group, probably due to a longer call duration caused by a deliberate better attention to the customer, but without degrading any other KPI. With respect to the perceived benefits of the treatment, the questionnaires show relevant improvements perceived by most employees after practicing mindfulness.Conclusions: We confirm that mindfulness improves attention awareness and benefits the working and personal life of helpdesk employees. However, further research is needed to identify a clear impact on productivity.

Context.In software engineering there is a widespread assumption that experience improves requirements analyst effectiveness, although empirical studies demonstrate the opposite.Aim.Determine whether experience (interviews, eliciting, development, professional) influences requirements elicitation using interviews.Method.We ran 12 quasi-experiments recruiting 124 subjects in which we measured analyst effectiveness as the number of items (i.e., concepts, rules, processes) correctly elicited. The experimental task was to elicit requirements using the open interview technique followed by the consolidation of the elicited information in domains with which the analysts were and were not familiar.Results.In unfamiliar domains, interview experience, requirements experience, development experience, and professional experience does not have any relationship with analyst effectiveness. In familiar domains, effectiveness varies depending on the type of experience. Interview experience has a positive effect, whereas professional experience has a moderate negative effect. Requirements experience appears to have a moderately positive effect; however, the statistical power of the analysis is insufficient to be able to confirm this point. Development experience has no effect.Conclusion.Experience impacts analyst effectiveness differently depending on the problem domain type (familiar, unfamiliar). Generally, experience does not account for all the observed variability in effectiveness, so there are other influential factors.

Context: The usability software quality characteristic aims to improve system user performance. In a previous study, we found evidence of the impact of a set of usability features from the viewpoint of users in terms of efficiency, effectiveness and satisfaction. However, the impact level appears to depend on the usability feature and suggest priorities with respect to their implementation depending on how they promote user performance.Objectives: We use a family of three experiments to increase the precision and generalization of the results in the baseline experiment and provide findings regarding the impact on user performance of the Abort Operation, Progress Feedback and Preferences usability mechanisms.Method: We conduct two replications of the baseline experiment in academic settings. We analyse the data of 366 experimental subjects and apply aggregation (meta-analysis) procedures.Results: We find that the Abort Operation and Preferences usability mechanisms appear to improve system usability a great deal with respect to efficiency, effectiveness and user satisfaction.Conclusions: We find that the family of experiments further corroborates the results of the baseline experiment. Most of the results are statistically significant, and, because of the large number of experimental subjects, the evidence that we gathered in the replications is sufficient to outweigh other experiments.

The significant momentum and importance of Mining Software Repositories (MSR) in Software Engineering (SE) has fostered new opportunities and challenges for extensive empirical research. However, MSR researchers seem to struggle to characterize the empirical methods they use into the existing empirical SE body of knowledge. This is especially the case of MSR experiments. To provide evidence on the special characteristics of MSR experiments and their differences with experiments traditionally acknowledged in SE so far, we elicited the hallmarks that differentiate an experiment from other types of empirical studies and characterized the hallmarks and types of experiments in MSR. We analyzed MSR literature obtained from a small-scale systematic mapping study to assess the use of the term experiment in MSR. We found that 19% of the papers claiming to be an experiment are indeed not an experiment at all but also observational studies, so they use the term in a misleading way. From the remaining 81% of the papers, only one of them refers to a genuine controlled experiment while the others stand for experiments with limited control. MSR researchers tend to overlook such limitations, compromising the interpretation of the results of their studies. We provide recommendations and insights to support the improvement of MSR experiments.

Context. Mindfulness is a meditation technique whose main goal is keeping the mind calm and educating attention by focusing only on one thing at a time, usually breathing. The reported benefits of its continued practice can be of interest for Software Engineering students and practitioners, especially in tasks like conceptual modeling, in which concentration and clearness of mind are crucial.Goal. In order to evaluate whether Software Engineering students enhance their conceptual modeling performance after several weeks of mindfulness practice, a series of three controlled experiments were carried out at the University of Seville during three consecutive academic years (2013–2016) involving 130 students.Method. In all the experiments, the subjects were divided into two groups. While the experimental group practiced mindfulness, the control group was trained in public speaking as a placebo treatment. All the subjects developed two conceptual models based on a transcript of an interview, one before and another one after the treatment. The results were compared in terms of conceptual modeling quality (measured as effectiveness, i.e., the percentage of model elements correctly identified) and productivity (measured as efficiency, i.e., the number of model elements correctly identified per unit of time).Results. The statistically significant results of the series of experiments revealed that the subjects who practiced mindfulness developed slightly better conceptual models (their quality was 8.16 percent higher) and they did it faster (they were 46.67 percent more productive) than the control group, even if they did not have a previous interest in meditation.Conclusions. The practice of mindfulness improves the performance of Software Engineering students in conceptual modeling, especially their productivity. Nevertheless, more experimentation is needed in order to confirm the outcomes in other Software Engineering tasks and populations.

Background: Test-Driven Development (TDD) is an iterative software development process characterized by test-code-refactor cycle. TDD recommends that developers work on small and manageable tasks at each iteration. However, the ability to break tasks into small work items effectively is a learned skill that improves with experience. In experimental studies of TDD, the granularity of task descriptions is an overlooked factor. In particular, providing a more granular task description in terms of a set of sub-tasks versus providing a coarser-grained, generic description. Objective: We aim to investigate the impact of task description granularity on the outcome of TDD, as implemented by novice developers, with respect to software quality, as measured by functional correctness and functional completeness. Method: We conducted a one-factor crossover experiment with 48 graduate students in an academic environment. Each participant applied TDD and implemented two tasks, where one of the tasks was presented using a more granular task description. Resulting artifacts were evaluated with acceptance tests to assess functional correctness and functional completeness. Linear mixed-effects models (LMM) were used for analysis. Results: Software quality improved significantly when participants applied TDD using more granular task descriptions. The effect of task description granularity is statistically significant and had a medium to large effect size. Moreover, the task was found to be a significant predictor of software quality which is an interesting result (because two tasks used in the experiment were considered to be of similar complexity). Conclusion: For novice TDD practitioners, the outcome of TDD is highly coupled with the ability to break down the task into smaller parts. For researchers, task selection and task description granularity requires more attention in the design of TDD experiments. Task description granularity should be taken into account in secondary studies. Further comparative studies are needed to investigate whether task descriptions affect other development processes similarly.

Context: There is a lack of empirical evidence on the differences between model-driven development (MDD), where code is automatically derived from conceptual models, and traditional software development method, where code is manually written. In our previous work, we compared both methods in a baseline experiment concluding that quality of the software developed following MDD was significantly better only for more complex problems (with more function points). Quality was measured through test cases run on a functional system. Objective: This paper reports six replications of the baseline to study the impact of problem complexity on software quality in the context of MDD. Method: We conducted replications of two types: strict replications and object replications. Strict replications were similar to the baseline, whereas we used more complex experimental objects (problems) in the object replications. Results: MDD yields better quality independently of problem complexity with a moderate effect size. This effect is bigger for problems that are more complex. Conclusions: Thanks to the bigger size of the sample after aggregating replications, we discovered an effect that the baseline had not revealed due to the small sample size. The baseline results hold, which suggests that MDD yields better quality for more complex problems.

Context: Researchers from different groups and institutions are collaborating on building groups of experiments by means of replication (i.e., conducting groups of replications). Disparate aggregation techniques are being applied to analyze groups of replications. The application of unsuitable techniques to aggregate replication results may undermine the potential of groups of replications to provide in-depth insights from experiment results. Objectives: Provide an analysis procedure with a set of embedded guidelines to aggregate software engineering (SE) replication results. Method: We compare the characteristics of groups of replications for SE and other mature experimental disciplines such as medicine and pharmacology. In view of their differences, the limitations with regard to the joint data analysis of groups of SE replications and the guidelines provided in mature experimental disciplines to analyze groups of replications, we build an analysis procedure with a set of embedded guidelines specifically tailored to the analysis of groups of SE replications. We apply the proposed analysis procedure to a representative group of SE replications to illustrate its use. Results: All the information contained within the raw data should be leveraged during the aggregation of replication results. The analysis procedure that we propose encourages the use of stratified individual participant data and aggregated data in tandem to analyze groups of SE replications. Conclusion: The aggregation techniques used to analyze groups of replications should be justified in research articles. This will increase the reliability and transparency of joint results. The proposed guidelines should ease this endeavor.

Reviews on test-driven development (TDD) studies suggest that the conflicting results reported in the literature are due to unobserved factors, such as the tasks used in the experiments, and highlight that there are very few industry experiments conducted with professionals. The goal of this study is to investigate the impact of a new factor, the chosentask, and thedevelopment approachon external quality in an industrial experimental setting with 17 professionals. The participants are junior to senior developers in programming with Java, beginner to novice in unit testing, JUnit, and they have no prior experience in TDD. The experimental design is a$2\times 2$cross-over, i.e., we use two tasks for each of the two approaches, namely TDD and incremental test-last development (ITLD). Our results reveal that bothdevelopment approachandtaskare significant factors with regards to the external quality achieved by the participants. More specifically, the participants produce higher quality code during ITLD in which splitting user stories into subtasks, coding, and testing activities are followed, compared to TDD. The results also indicate that the participants produce higher quality code during the implementation of Bowling Score Keeper, compared to that of Mars Rover API, although they perceived both tasks as of similar complexity. An interaction between thedevelopment approachandtaskcould not be observed in this experiment. We conclude that variables that have not been explored so often, such as the extent to which the task is specified in terms of smaller subtasks, and developers’ unit testing experience might be critical factors in TDD experiments. The real-world appliance of TDD and its implications on external quality still remain to be challenging unless these uncontrolled and unconsidered factors are further investigated by researchers in both academic and industrial settings.

Background: Publication bias is the failure to publish the results of a study based on the direction or strength of the study findings. The existence of publication bias is firmly established in areas like medical research. Recent research suggests the existence of publication bias in Software Engineering. Aims: Finding out whether experiments published in the International Workshop on Empirical Software Engineering and Measurement (ESEM) are affected by publication bias. Method: We review experiments published in ESEM. We also survey with experimental researchers to triangulate our findings. Results: ESEM experiments do not define hypotheses and frequently perform multiple testing. One-tailed tests have a slightly higher rate of achieving statistically significant results. We could not find other practices associated with publication bias. Conclusions: Our results provide a more encouraging perspective of SE research than previous research: (1) ESEM publications do not seem to be strongly affected by biases and (2) we identify some practices that could be associated with p-hacking, but it is more likely that they are related to the conduction of exploratory research.

Background. Most Mining Software Repositories (MSR) studies cannot obtain causal relations because they are not controlled experiments. The use of cohort studies as defined in epidemiology could help to overcome this shortcoming.

Researcher Bias (RB) occurs when researchers influence the results of an empirical study based on their expectations. RB might be due to the use of Questionable Research Practices (QRPs). In research fields like medicine, blinding techniques have been applied to counteract RB. We conducted an explorative qualitative survey to investigate RB in Software Engineering (SE) experiments, with respect to: (i) QRPs potentially leading to RB, (ii) causes behind RB, and (iii) possible actions to counteract RB including blinding techniques. Data collection was based on semi-structured interviews. We interviewed nine active experts in the empirical SE community. We then analyzed the transcripts of these interviews through thematic analysis. We found that some QRPs are acceptable in certain cases. Also, it appears that the presence of RB is perceived in SE and, to counteract RB, a number of solutions have been highlighted: some are intended for SE researchers and others for the boards of SE research outlets.

Abstract Software engineering (SE) experiments suffer from threats to validity that may impact their results. Replication allows researchers building on top of previous experiments’ weaknesses and increasing the reliability of the findings. Illustrating the benefits of replication to increase the reliability of the findings and uncover moderator variables. We replicate an experiment on test-driven development (TDD) and address some of its threats to validity and those of a previous replication. We compare the replications’ results and hypothesize on plausible moderators impacting results. Differences across TDD replications’ results might be due to the operationalization of the response variables, the allocation of subjects to treatments, the allowance to work outside the laboratory, the provision of stubs, or the task. Replications allow examining the robustness of the findings, hypothesizing on plausible moderators influencing results, and strengthening the evidence obtained.

We present a quasi-experiment to investigate whether, and to what extent, sleep deprivation impacts the performance of novice software developers using the agile practice of test-first development (TFD). We recruited 45 undergraduates, and asked them to tackle a programming task. Among the participants, 23 agreed to stay awake the night before carrying out the task, while 22 slept normally. We analyzed the quality (i.e., the functional correctness) of the implementations delivered by the participants in both groups, their engagement in writing source code (i.e., the amount of activities performed in the IDE while tackling the programming task) and ability to apply TFD (i.e., the extent to which a participant is able to apply this practice). By comparing the two groups of participants, we found that a single night of sleep deprivation leads to a reduction of 50 percent in the quality of the implementations. There is notable evidence that the developers' engagement and their prowess to apply TFD are negatively impacted. Our results also show that sleep-deprived developers make more fixes to syntactic mistakes in the source code. We conclude that sleep deprivation has possibly disruptive effects on software development activities. The results open opportunities for improving developers' performance by integrating the study of sleep with other psycho-physiological factors in which the software engineering research community has recently taken an interest in.

Context: Families of experiments (i.e., groups of experiments with the same goal) are on the rise in Software Engineering (SE). Selecting unsuitable aggregation techniques to analyze families may undermine their potential to provide in-depth insights from experiments' results. Objectives: Identifying the techniques used to aggregate experiments' results within families in SE. Raising awareness of the importance of applying suitable aggregation techniques to reach reliable conclusions within families. Method: We conduct a systematic mapping study (SMS) to identify the aggregation techniques used to analyze families of experiments in SE. We outline the advantages and disadvantages of each aggregation technique according to mature experimental disciplines such as medicine and pharmacology. We provide preliminary recommendations to analyze and report families of experiments in view of families' common limitations with regard to joint data analysis. Results: Several aggregation techniques have been used to analyze SE families of experiments, including Narrative synthesis, Aggregated Data (AD), Individual Participant Data (IPD) mega-trial or stratified, and Aggregation of p-values. The rationale used to select aggregation techniques is rarely discussed within families. Families of experiments are commonly analyzed with unsuitable aggregation techniques according to the literature of mature experimental disciplines. Conclusion: Data analysis' reporting practices should be improved to increase the reliability and transparency of joint results. AD and IPD stratified appear to be suitable to analyze SE families of experiments.

Test-Driven Development (TDD), an agile development approach that enforces the construction of software systems by means of successive micro-iterative testing coding cycles, has been widely claimed to increase external software quality. In view of this, some managers at Paf-a Nordic gaming entertainment company-were interested in knowing how would TDD perform at their premises. Eventually, if TDD outperformed their traditional way of coding (i.e., YW, short for Your Way), it would be possible to switch to TDD considering the empirical evidence achieved at the company level. We conduct an experiment at Paf to evaluate the performance of TDD, YW and the reverse approach of TDD (i.e., ITL, short for Iterative-Test Last) on external quality. TDD outperforms YW and ITL at Paf. Despite the encouraging results, we cannot recommend Paf to immediately adopt TDD as the difference in performance between YW and TDD is small. However, as TDD looks promising at Paf, we suggest to move some developers to TDD and to run a future experiment to compare the performance of TDD and YW. TDD slightly outperforms ITL in controlled experiments for TDD novices. However, more industrial experiments are still needed to evaluate the performance of TDD in real-life contexts.

Background: Noise, defined as an unwanted sound, is one of the commonest factors that could affect people's performance in their daily work activities. The software engineering research community has marginally investigated the effects of noise on software engineers' performance.

Background: Test-Driven Development (TDD) is an agile software development practice, which is claimed to boost both external quality of software products and developers' productivity.

Context: Researchers from different groups and institutions are collaborating towards the construction of groups of interrelated replications. Applying unsuitable techniques to aggregate interrelated replications' results may impact the reliability of joint conclusions.

Background: Statistical concepts and techniques are often applied incorrectly, even in mature disciplines such as medicine or psychology. Surprisingly, there are very few works that study statistical problems in software engineering (SE). Aim: Assess the existence of statistical errors in SE experiments. Method: Compile the most common statistical errors in experimental disciplines. Survey experiments published in ICSE to assess whether errors occur in high quality SE publications. Results: The same errors as identified in others disciplines were found in ICSE experiments, where 30% of the reviewed papers included several error types such as: a) missing statistical hypotheses, b) missing sample size calculation, c) failure to assess statistical test assumptions, and d) uncorrected multiple testing. This rather large error rate is greater for research papers where experiments are confined to the validation section. The origin of the errors can be traced back to: a) researchers not having sufficient statistical training, and, b) a profusion of exploratory research. Conclusions: This paper provides preliminary evidence that SE research suffers from the same statistical problems as other experimental disciplines. However, the SE community appears to be unaware of any shortcomings in its experiments, whereas other disciplines work hard to avoid these threats. Further research is necessary to find the underlying causes and set up corrective measures, but there are some potentially effective actions and are a priori easy to implement: a) improve the statistical training of SE researchers, and b) enforce quality assessment and reporting guidelines in SE publications.

This extended abstract summarizes an article, which has been published in the Empirical Software Engineering Journal and was selected for the Journal-First presentations at the International Conference on Software and System Process (ICSSP 2018).

Background: Writing unit tests is one of the primary activities in test-driven development. Yet, the existing reviews report few evidence supporting or refuting the effect of this development approach on test case quality. Lack of ability and skills of developers to produce sufficiently good test cases are also reported as limitations of applying test-driven development in industrial practice. Objective: We investigate the impact of test-driven development on the effectiveness of unit test cases compared to an incremental test last development in an industrial context. Method: We conducted an experiment in an industrial setting with 24 professionals. Professionals followed the two development approaches to implement the tasks. We measure unit test effectiveness in terms of mutation score. We also measure branch and method coverage of test suites to compare our results with the literature. Results: In terms of mutation score, we have found that the test cases written for a test-driven development task have a higher defect detection ability than test cases written for an incremental test-last development task. Subjects wrote test cases that cover more branches on a test-driven development task compared to the other task. However, test cases written for an incremental test-last development task cover more methods than those written for the second task. Conclusion: Our findings are different from previous studies conducted at academic settings. Professionals were able to perform more effective unit testing with test-driven development. Furthermore, we observe that the coverage measure preferred in academic studies reveal different aspects of a development approach. Our results need to be validated in larger industrial contexts.

Background: Test-driven development (TDD) is a technique that repeats short coding cycles interleaved with testing. The developer first writes a unit test for the desired functionality, followed by the necessary production code, and refactors the code. Many empirical studies neglect unique process characteristics related to TDD iterative nature. Aim: We formulate four process characteristic: sequencing, granularity, uniformity, and refactoring effort. We investigate how these characteristics impact quality and productivity in TDD and related variations. Method: We analyzed 82 data points collected from 39 professionals, each capturing the process used while performing a specific development task. We built regression models to assess the impact of process characteristics on quality and productivity. Quality was measured by functional correctness. Result: Quality and productivity improvements were primarily positively associated with the granularity and uniformity. Sequencing, the order in which test and production code are written, had no important influence. Refactoring effort was negatively associated with both outcomes. We explain the unexpected negative correlation with quality by possible prevalence of mixed refactoring. Conclusion: The claimed benefits of TDD may not be due to its distinctive test-first dynamic, but rather due to the fact that TDD-like processes encourage fine-grained, steady steps that improve focus and flow.

Context: Test-driven development (TDD) is an agile practice claimed to improve the quality of a software product, as well as the productivity of its developers. A previous study (i.e., baseline experiment) at the University of Oulu (Finland) compared TDD to a test-last development (TLD) approach through a randomized controlled trial. The results failed to support the claims. Goal: We want to validate the original study results by replicating it at the University of Basilicata (Italy), using a different design. Method: We replicated the baseline experiment, using a crossover design, with 21 graduate students. We kept the settings and context as close as possible to the baseline experiment. In order to limit researchers bias, we involved two other sites (UPM, Spain, and Brunel, UK) to conduct blind analysis of the data. Results: The Kruskal-Wallis tests did not show any significant difference between TDD and TLD in terms of testing effort (p-value = .27), external code quality (p-value = .82), and developers' productivity (p-value = .83). Nevertheless, our data revealed a difference based on the order in which TDD and TLD were applied, though no carry over effect. Conclusions: We verify the baseline study results, yet our results raises concerns regarding the selection of experimental objects, particularly with respect to their interaction with the order in which of treatments are applied.

Context. Requirements elicitation is a highly communicative activity in which human interactions play a critical role. A number of analyst characteristics or skills may influence elicitation process effectiveness. Aim. Study the influence of analyst problem domain knowledge on elicitation effectiveness. Method. We executed a controlled experiment with post-graduate students. The experimental task was to elicit requirements using open interview and consolidate the elicited information immediately afterwards. We used four different problem domains about which students had different levels of knowledge. Two tasks were used in the experiment, whereas the other two were used in an internal replication of the experiment; that is, we repeated the experiment with the same subjects but with different domains. Results. Analyst problem domain knowledge has a small but statistically significant effect on the effectiveness of the requirements elicitation activity. The interviewee has a big positive and significant influence, as does general training in requirements activities and interview experience. Conclusion. During early contacts with the customer, a key factor is the interviewee; however, training in tasks related to requirements elicitation and knowledge of the problem domain helps requirements analysts to be more effective.

Context.Several empirical studies have explored the benefits of software design patterns, but their collective results are highly inconsistent. Resolving the inconsistencies requires investigating moderators—i.e., variables that cause an effect to differ across contexts.Objectives.Replicate a design patterns experiment at multiple sites and identify sufficient moderators to generalize the results across prior studies.Methods.We perform a close replication of an experiment investigating the impact (in terms of time and quality) of design patterns (Decorator and Abstract Factory) on software maintenance. The experiment was replicated once previously, with divergent results. We execute our replication at four universities—spanning two continents and three countries—using a new method for performing distributed replications based on closely coordinated, small-scale instances (“joint replication”). We perform two analyses: 1) apost-hocanalysis of moderators, based on frequentist and Bayesian statistics; 2) ana priorianalysis of the original hypotheses, based on frequentist statistics.Results.The main effect differs across the previous instances of the experiment and across the sites in our distributed replication. Our analysis of moderators (including developer experience and pattern knowledge) resolves the differences sufficiently to allow for cross-context (and cross-study) conclusions. The final conclusions represent 126 participants from five universities and 12 software companies, spanning two continents and at least four countries.Conclusions.The Decorator pattern is found to be preferable to a simpler solution during maintenance, as long as the developer has at least some prior knowledge of the pattern. For Abstract Factory, the simpler solution is found to be mostly equivalent to the pattern solution. Abstract Factory is shown to require a higher level of knowledge and/or experience than Decorator for the pattern to be beneficial.

In experiments with crossover design subjects apply more than one treatment. Crossover designs are widespread in software engineering experimentation: they require fewer subjects and control the variability among subjects. However, some researchers disapprove of crossover designs. The main criticisms are: the carryover threat and its troublesome analysis. Carryover is the persistence of the effect of one treatment when another treatment is applied later. It may invalidate the results of an experiment. Additionally, crossover designs are often not properly designed and/or analysed, limiting the validity of the results. In this paper, we aim to make SE researchers aware of the perils of crossover experiments and provide risk avoidance good practices. We study how another discipline (medicine) runs crossover experiments. We review the SE literature and discuss which good practices tend not to be adhered to, giving advice on how they should be applied in SE experiments. We illustrate the concepts discussed analysing a crossover experiment that we have run. We conclude that crossover experiments can yield valid results, provided they are properly designed and analysed, and that, if correctly addressed, carryover is no worse than other validity threats.

Background: Most of the experiments in software engineering (SE) employ students as subjects. This raises concerns about the realism of the results acquired through students and adaptability of the results to software industry. Aim: We compare students and professionals to understand how well students represent professionals as experimental subjects in SE research. Method: The comparison was made in the context of two test-driven development experiments conducted with students in an academic setting and with professionals in a software organization. We measured the code quality of several tasks implemented by both subject groups and checked whether students and professionals perform similarly in terms of code quality metrics. Results: Except for minor differences, neither of the subject groups is better than the other. Professionals produce larger, yet less complex, methods when they use their traditional development approach, whereas both subject groups perform similarly when they apply a new approach for the first time. Conclusion: Given a carefully scoped experiment on a development approach that is new to both students and professionals, similar performances are observed. Further investigation is necessary to analyze the effects of subject demographics and level of experience on the results of SE experiments.

Usability is a software system quality attribute. Although software engineers originally considered usability to be related exclusively to the user interface, it was later found to affect the core functionality of software applications. As of then, proposals for addressing usability at different stages of the software development cycle were researched. The objective of this paper is to present three reusable solutions at detailed design and programming level in order to effectively implement the Abort Operation, Progress Feedback and Preferences usability functionalities in web applications. To do this, an inductive research method was applied. We developed three web applications including the above usability functionalities as case studies. We looked for commonalities across the implementations in order to induce a general solution. The elements common to all three developed applications include: application scenarios, functionalities, responsibilities, classes, methods, attributes and code snippets. The findings were specified as an implementation-oriented design pattern and as programming patterns in three languages. Additional case studies were conducted in order to validate the proposed solution. The independent developers used the patterns to implement different applications for each case study. As a result, we found that solutions specified as patterns can be reused to develop web applications.

Context: Measurement is crucial and important to empirical software engineering. Although reliability and validity are two important properties warranting consideration in measurement processes, they may be influenced by random or systematic error (bias) depending on which metric is used. Aim: Check whether, the simple subjective metrics used in empirical software engineering studies are prone to bias. Method: Comparison of the reliability of a family of empirical studies on requirements elicitation that explore the same phenomenon using different design types and objective and subjective metrics. Results: The objectively measured variables (experience and knowledge) tend to achieve more reliable results, whereas subjective metrics using Likert scales (expertise and familiarity) tend to be influenced by systematic error or bias. Conclusions: Studies that predominantly use variables measured subjectively, like opinion polls or expert opinion acquisition, must take every care to prevent bias that can result in incorrect results.

Context: The software engineering community is becoming more aware of the need for experimental replications. In spite of the importance of this topic, there is still much inconsistency in the terminology used to describe replications.

Context: Empirical Software Engineering (ESE) replication researchers need to store and manipulate experimental data for several purposes, in particular analysis and reporting. Current research needs call for sharing and preservation of experimental data as well. In a previous work, we analyzed Replication Data Management (RDM) needs. A novel concept, called Experimental Ecosystem, was proposed to solve current deficiencies in RDM approaches. The empirical ecosystem provides replication researchers with a common framework that integrates transparently local heterogeneous data sources. A typical situation where the Empirical Ecosystem is applicable, is when several members of a research group, or several research groups collaborating together, need to share and access each other experimental results. However, to be able to apply the Empirical Ecosystem concept and deliver all promised benefits, it is necessary to analyze the software architectures and tools that can properly support it.

Context: Juristo et al. [7] published a literature review about testing technique experiments. The goal was to provide a picture of which techniques and aspects of techniques had been studied experimentally, and try to compile a body of knowledge on testing techniques. Goal: In this paper, we extend Juristo et al.'s study to cover the years from 2000 (where it ended) until 2013. Method: We have performed a systematic mapping study. Results: The situation in testing experimentation has not changed since Juristo et al.'s study. Conclusions: The research field has the same shortcomings.

[Context] Replication Data Management (RDM) aims at enabling the use of data collections from several itera-tions of an experiment. However, there are several major chal-lenges to RDM from integrating data models and data from em-pirical study infrastructures that were not designed to cooperate, e.g., data model variation of local data sources. [Objective] In this paper we analyze RDM needs and evaluate conceptual RDM approaches to support replication researchers. [Method] We adapted the ATAM evaluation process to (a) analyze RDM use cases and needs of empirical replication study research groups and (b) compare three conceptual approaches to address these RDM needs: central data repositories with a fixed data model, heterogeneous local repositories, and an empirical ecosystem. [Results] While the central and local approaches have major issues that are hard to resolve in practice, the empirical ecosys-tem allows bridging current gaps in RDM from heterogeneous data sources. [Conclusions] The empirical ecosystem approach should be explored in diverse empirical environments.

Summary form only given. To consolidate a body of knowledge built upon evidence, experimental results have to be extensively verified. Experiments need replication at other times and under other conditions before they can produce an established piece of knowledge. Several replications need to be run to strengthen the evidence. Most SE experiments have not been replicated. If an experiment is not replicated, there is no way to distinguish whether results were produced by chance (the observed event occurred accidentally), results are artifactual (the event occurred because of the experimental configuration but does not exist in reality) or results conform to a pattern existing in reality. The immaturity of experimental SE knowledge has been an obstacle to replication. Context differences usually oblige SE experimenters to adapt experiments for replication. As key experimental conditions are yet unknown, slight changes in replications have led to differences in the results that prevent verification. There are still many uncertainties about how to proceed with replications of SE experiments. Should replicators reuse the baseline experiment materials? How much liaison should there be among the original and replicating experimenters, if any? What elements of the experimental configuration can be changed for the experiment to be considered a replication rather than a new experiment? The aim of replication is to verify results, but different types of replication serve special verification purposes and afford different degrees of change. Each replication type helps to discover particular experimental conditions that might influence the results. We need to learn which types of replications are feasible in SE as well as the acceptable changes for each type and the level of verification provided.

Abstract?Background: There is no globally accepted open source software development process to define how open source software is developed in practice. A process description is important for coordinating all the software development activities involving both people and technology. Aim: The research question that this study sets out to answer is: What activities do open source software process models contain? The activity groups on which it focuses are Concept Exploration, Software Requirements, Design, Maintenance and Evaluation. Method: We conduct a systematic mapping study (SMS). A SMS is a form of systematic literature review that aims to identify and classify available research papers concerning a particular issue. Results: We located a total of 29 primary studies, which we categorized by the open source software project that they examine and by activity types (Concept Exploration, Software Requirements, Design, Maintenance and Evaluation). The activities present in most of the open source software development processes were Execute Tests and Conduct Reviews, which belong to the Evaluation activities group. Maintenance is the only group that has primary studies addressing all the activities that it contains. Conclusions: The primary studies located by the SMS are the starting point for analyzing the open source software development process and proposing a process model for this community. The papers in our paper pool that describe a specific open source software project provide more regarding our research question than the papers that talk about open source software development without referring to a specific open source software project.

Some verification and validation techniques have been evaluated both theoretically and empirically. Most empirical studies have been conducted without subjects, passing over any effect testers have when they apply the techniques. We have run an experiment with students to evaluate the effectiveness of three verification and validation techniques (equivalence partitioning, branch testing and code reading by stepwise abstraction). We have studied how well able the techniques are to reveal defects in three programs. We have replicated the experiment eight times at different sites. Our results show that equivalence partitioning and branch testing are equally effective and better than code reading by stepwise abstraction. The effectiveness of code reading by stepwise abstraction varies significantly from program to program. Finally, we have identified project contextual variables that should be considered when applying any verification and validation technique or to choose one particular technique.

Background: Several meta-analysis methods can be used to quantitatively combine the results of a group of experiments, including the weighted mean difference, statistical vote counting, the parametric response ratio and the non-parametric response ratio. The software engineering community has focused on the weighted mean difference method. However, other meta-analysis methods have distinct strengths, such as being able to be used when variances are not reported. There are as yet no guidelines to indicate which method is best for use in each case Aim: Compile a set of rules that SE researchers can use to ascertain which aggregation method is best for use in the synthesis phase of a systematic review. Method: Monte Carlo simulation varying the number of experiments in the meta analyses, the number of subjects that they include, their variance and effect size. We empirically calculated the reliability and statistical power in each case Results: WMD is generally reliable if the variance is low, whereas its power depends on the effect size and number of subjects per meta-analysis; the reliability of RR is generally unaffected by changes in variance, but it does require more subjects than WMD to be powerful; NPRR is the most reliable method, but it is not very powerful; SVC behaves well when the effect size is moderate, but is less reliable with other effect sizes. Detailed tables of results are annexed. Conclusions: Before undertaking statistical aggregation in software engineering, it is worthwhile checking whether there is any appreciable difference in the reliability and power of the methods. If there is, software engineers should select the method that optimizes both parameters.

All meta-analyses should include a heterogeneity analysis. Even so, it is not easy to decide whether a set of studies are homogeneous or heterogeneous because of the low statistical power of the statistics used (usually the Q test). Objective: Determine a set of rules enabling SE researchers to find out, based on the characteristics of the experiments to be aggregated, whether or not it is feasible to accurately detect heterogeneity. Method: Evaluate the statistical power of heterogeneity detection methods using a Monte Carlo simulation process. Results: The Q test is not powerful when the meta-analysis contains up to a total of about 200 experimental subjects and the effect size difference is less than 1. Conclusions: The Q test cannot be used as a decision-making criterion for meta-analysis in small sample settings like SE. Random effects models should be used instead of fixed effects models. Caution should be exercised when applying Q test-mediated decomposition into subgroups.

Quality assessment is one of the activities performed as part of systematic literature reviews. It is commonly accepted that a good quality experiment is bias free. Bias is considered to be related to internal validity (e.g., how adequately the experiment is planned, executed and analysed). Quality assessment is usually conducted using checklists and quality scales. It has not yet been proven, however, that quality is related to experimental bias. Aim: Identify whether there is a relationship between internal validity and bias in software engineering experiments. Method: We built a quality scale to determine the quality of the studies, which we applied to 28 experiments included in two systematic literature reviews. We proposed an objective indicator of experimental bias, which we applied to the same 28 experiments. Finally, we analysed the correlations between the quality scores and the proposed measure of bias. Results: We failed to find a relationship between the global quality score (resulting from the quality scale) and bias, however, we did identify interesting correlations between bias and some particular aspects of internal validity measured by the instrument. Conclusions: There is an empirically provable relationship between internal validity and bias. It is feasible to apply quality assessment in systematic literature reviews, subject to limits on the internal validity aspects for consideration.

We have located the results of empirical studies on elicitation techniques and aggregated these results to gather empirically grounded evidence. Our chosen surveying methodology was systematic review, whereas we used an adaptation of comparative analysis for aggregation because meta-analysis techniques could not be applied. The review identified 564 publications from the SCOPUS, IEEEXPLORE, and ACM DL databases, as well as Google. We selected and extracted data from 26 of those publications. The selected publications contain 30 empirical studies. These studies were designed to test 43 elicitation techniques and 50 different response variables. We got 100 separate results from the experiments. The aggregation generated 17 pieces of knowledge about the interviewing, laddering, sorting, and protocol analysis elicitation techniques. We provide a set of guidelines based on the gathered pieces of knowledge.

Experiment replication is a key component of the scientific paradigm. The purpose of replication is to verify previously observed findings. Although some Software Engineering (SE) experiments have been replicated, yet, there is still disagreement about how replications should be run in our field. With the aim of gaining a better understanding of how replications are carried out, this paper examines different replication types in other scientific disciplines. We believe that by analysing the replication types proposed in other disciplines it is possible to clarify some of the question marks still hanging over experimental SE replication.

The RESER 2010 workshop provides a venue in which empirical Software Engineering researchers may present and discuss theoretical foundations and methods of replication, as well as the results of replicated studies.

In no science or engineering discipline does it make sense to speak of isolated experiments. The results of a single experiment cannot be viewed as representative of the underlying reality. The concept of experiment is closely related to replication. Experiment replication is the repetition of an experiment to double-check its results. Multiple replications of an experiment increase the credibility of its results. Software engineering has tried its hand at the identical repetition of experiments in the way of the natural sciences (physics, chemistry, etc.). After numerous attempts over the years, excepting experiments repeated by the same researchers at the same site, no exact replications have yet been achieved. One key reason for this is the complexity of the software development setting. This complexity prevents the many experimental conditions from being reproduced identically. This paper reports research into whether non-exact replications can be of any use. We propose a process that allows researchers to generate new knowledge when running non-exact replications. To illustrate the advantages of the proposed process, two different replications of an experiment are shown.

In no science or engineering discipline does it make sense to speak of isolated experiments. The results of a single experiment cannot be viewed as representative of the underlying reality. The concept of experiment is closely related to replication. Experiment replication is the repetition of an experiment to double-check its results. Multiple replications of an experiment increase the credibility of its results. Software engineering has tried its hand at the identical repetition of experiments in the way of the natural sciences (physics, chemistry, etc.). After numerous attempts over the years, excepting experiments repeated by the same researchers at the same site, no exact replications have yet been achieved. One key reason for this is the complexity of the software development setting. This complexity prevents the many experimental conditions from being reproduced identically. This paper reports research into whether non-exact replications can be of any use. We propose a process that allows researchers to generate new knowledge when running non-exact replications. To illustrate the advantages of the proposed process, two different replications of an experiment are shown.

Classification makes a significant contribution to advancing knowledge in both science and engineering. It is a way of investigating the relationships between the objects to be classified and identifies gaps in knowledge. Classification in engineering also has a practical application; it supports object selection. They can help mature software engineering knowledge, as classifications constitute an organized structure of knowledge items. Till date, there have been few attempts at classifying in software engineering. In this research, we examine how useful classifications in software engineering are for advancing knowledge by trying to classify testing techniques. The paper presents a preliminary classification of a set of unit testing techniques. To obtain this classification, we enacted a generic process for developing useful software engineering classifications. The proposed classification has been proven useful for maturing knowledge about testing techniques, and therefore, SE, as it helps to: 1) provide a systematic description of the techniques, 2) understand testing techniques by studying the relationships among techniques (measured in terms of differences and similarities), 3) identify potentially useful techniques that do not yet exist by analyzing gaps in the classification, and 4) support practitioners in testing technique selection by matching technique characteristics to project characteristics.

Like any other quality attribute, usability imposes specific constraints on software components. Features that raise the software system's usability have to be considered from the earliest development stages. But, discovering and documenting usability features is likely to be beyond the usability knowledge of most requirements engineers, developers, and users. We propose an approach based on developing specific guidelines that capitalize upon key elements recurrently intervening in the usability features elicitation and specification process. The use of these guidelines provides requirements analysts with a knowledge repository. They can use this repository to ask the right questions and capture precise usability requirements information.

Usability is increasingly recognized as a quality attribute that one has to explicitly deal with during development. Nevertheless, usability techniques, when applied, are decoupled from the software development process. The host of techniques offered by the HCI (Human-Computer Interaction) field make the task of selecting the most appropriate ones for a given project and organization a difficult task. Project managers and developers aiming to integrate usability practices into their software process have to face important challenges, as the techniques are not described in the frame of a software process as it is understood in SE (Software Engineering). Even when HCI experts (either in-house or from an external organization) are involved in the integration process, it is also a tough endeavour due to the strong differences in terminology and overall approach to software development between HCI and SE. In this tutorial we will present, from a SE viewpoint, which usability techniques can be most valuable to development teams with little or no previous usability experience, how a particular set of techniques can be selected according to the specific characteristics of the organization and project, and how usability techniques match with the activity groups in the development process.

This paper reports a systematic review of empirical studies concerning the effectiveness of elicitation techniques, and the subsequent aggregation of empirical evidence gathered from those studies. The most significant results of the aggregation process are as follows: (I) interviews, preferentially structured, appear to be one of the most effective elicitation techniques; (2) many techniques often cited in the literature, like card sorting, ranking or thinking aloud, tend to be less effective than interviews; (3) analyst experience does not appear to be a relevant factor; and (4) the studies conducted have not found the use of intermediate representations during elicitation to have significant positive effects. It should be noted that, as a general rule, the studies from which these results were aggregated have not been replicated, and therefore the above claims cannot be said to be absolutely certain. However, they can be used by researchers as pieces of knowledge to be further investigated and by practitioners in development projects, always taking into account that they are preliminary findings

Software architects have techniques to deal with many quality attributes such as performance, reliability, and maintainability. Usability, however, has traditionally been concerned primarily with presentation and not been a concern of software architects beyond separating the user interface from the remainder of the application. In this paper, we present usability-supporting architectural patterns. Each pattern describes a usability concern that is not supported by separation alone. For each concern, a usability-supporting architectural pattern provides the forces from the characteristics of the task and environment, the human, and the state of the software to motivate an implementation independent solution cast in terms of the responsibilities that must be fulfilled to satisfy the forces. Furthermore, each pattern includes a sample solution implemented in the context of an overriding separation based pattern such as J2EE Model View Controller.

Abstract This paper is based on the premise that people's behavioural competencies or characteristics of professional conduct influence the effectiveness and efficiency with which they perform a predetermined role in the software process. We propose a capabilities‐oriented process model that includes traditional elements of the software process (activities, products, techniques, people and roles) and the original element of this paper (capabilities). With the aim of adding behavioural competencies to the process model, we define the capability–person and capability–role relationships involved in software development. Additionally, we propose two procedures that are based on each of these relationships: a procedure that can be used to determine the capabilities of the members of a development team; and a procedure that can be used to assign people to perform roles depending on their capabilities and the capabilities demanded by the roles. Finally, the person–capabilities–role relationship has been empirically validated. The results yielded by this experiment confirm the hypothesis that assigning people to roles according to their capabilities and the capabilities demanded by the role improves software development. Copyright © 2004 John Wiley & Sons, Ltd.

Usability is increasingly recognized as a quality attribute that one has to design for. The conventional alternative is to measure usability on a finished system and improve it. The disadvantage of this approach is, obviously, that the cost associated with implementing usability improvements in a fully implemented system are typically very high and prohibit improvements with architectural impact. In this tutorial, we present the insights gained, techniques developed and lessons learned in the EU-IST project STATUS (SofTware Architectures That supports USability). These include a forward-engineering perspective on usability, a technique for specifying usability requirements, a method for assessing software architectures for usability and, finally, for improving software architectures for usability. The topics are extensively illustrated by examples and experiences from many industrial cases.

One of the main problems in software testing is the development of a suitable set of test cases so that the effectiveness of the test is maximised with a minimum number of test cases. A lot of testing techniques are now available for developing test cases. However, some of them are misused, others are never used and only a few are applied again and again. When developers have to decide what testing techniques(s) they should use in a project, they have little (if any) experiential information about the available testing techniques, their usefulness and, in general, how suited they are to the project. This paper presents the results of developing a characterization scheme for test technique selection. When instantiated for different techniques, the scheme should provide developers with enough information for choosing the best suited to their project. Thus, their decisions would be based on sound knowledge of the techniques, instead of perceptions, suppositions and assumptions.

This paper presents the motivations, experiences and results of teaching integrated Software Engineering (SE) and Knowledge Engineering (KE), specifically as part of the master course organized by the Polytechnic University of Madrid (School of Computer Science). The paper outlines a possible approach to this instruction, whose aim is for software practitioners thus educated to have a flexible and moldable view of the software systems development process. This broad and malleable approach allows future practitioners to better address the increasingly more complex, divergent and innovative problems and needs raised by users. This approach is the result of a gradual and continuous process. This paper discusses the current stage of integration, giving a detailed description and justification of the scope of the integrated instruction. For the purpose of quantitatively analyzing this experience, the paper also shows the results of the evaluation conducted throughout this process at three levels (industry, students and projects).

Software engineering (SE) and knowledge engineering (KE) develop software systems using different construction process models. Because of the growing complexity of the problems to be solved by computers, the conventional systems (CS) and knowledge-based systems (KBS) software process is at present passing through a period of integration. In this paper, we propose a software process model applicable to both CS and KBS. The model designed is declarative, that is, it indicates what is done to build a software system. Its goal is to provide software and knowledge engineers with a techno-conceptual tool to develop systems comprising both traditional and knowledge-based software.

The ten-year gap between the emergence of Software Engineering (SE) and Knowledge Engineering (KE) has led to the two disciplines developing along different methodological lines. In this paper, we point out that, after having passed through a period during which they ignored each other, followed by a competitive phase, the two disciplines have now reached a meeting point. We see the need for a life-cycle model for systems that integrate traditional and knowledge-based software. Besides, software development in the 21st century will entail open requirements and technological tools that will evolve during the life-cycle. Finally, the paper discusses a proposal for a conical-type spiral life-cycle model that seeks to meet all those needs.