Abstract Tech-leading organizations are embracing the forthcoming artificial intelligence revolution. Intelligent systems are replacing and cooperating with traditional software components. Thus, the same development processes and standards in software engineering ought to be complied in artificial intelligence systems. This study aims to understand the processes by which artificial intelligence-based systems are developed and how state-of-the-art lifecycle models fit the current needs of the industry. We conducted an exploratory case study at ING, a global bank with a strong European base. We interviewed 17 people with different roles and from different departments within the organization. We have found that the following stages have been overlooked by previous lifecycle models: data collection , feasibility study , documentation , model monitoring , and model risk assessment . Our work shows that the real challenges of applying Machine Learning go much beyond sophisticated learning algorithms – more focus is needed on the entire lifecycle. In particular, regardless of the existing development tools for Machine Learning, we observe that they are still not meeting the particularities of this field.

In 2014, a Microsoft study investigated the sort of questions that data science applied to software engineering should answer. This resulted in 145 questions that developers considered relevant for data scientists to answer, thus providing a research agenda to the community. Fast forward to five years, no further studies investigated whether the questions from the software engineers at Microsoft hold for other software companies, including software-intensive companies with different primary focus (to which we refer as software-defined enterprises). Furthermore, it is not evident that the problems identified five years ago are still applicable, given the technological advances in software engineering. This paper presents a study at ING, a software-defined enterprise in banking in which over 15,000 IT staff provides in-house software solutions. This paper presents a comprehensive guide of questions for data scientists selected from the previous study at Microsoft along with our current work at ING. We replicated the original Microsoft study at ING, looking for questions that impact both software companies and software-defined enterprises and continue to impact software engineering. We also add new questions that emerged from differences in the context of the two companies and the five years gap in between. Our results show that software engineering questions for data scientists in the software-defined enterprise are largely similar to the software company, albeit with exceptions. We hope that the software engineering research community builds on the new list of questions to create a useful body of knowledge.

The appeal of delivering new features faster has led many software projects to adopt rapid releases. However, it is not well understood what the effects of this practice are. This paper presents an exploratory case study of rapid releases at ING, a large banking company that develops software solutions in-house, to characterize rapid releases. Since 2011, ING has shifted to a rapid release model. This switch has resulted in a mixed environment of 611 teams releasing relatively fast and slow. We followed a mixed-methods approach in which we conducted a survey with 461 participants and corroborated their perceptions with 2 years of code quality data and 1 year of release delay data. Our research shows that: rapid releases are more commonly delayed than their non-rapid counterparts, however, rapid releases have shorter delays; rapid releases can be beneficial in terms of reviewing and user-perceived quality; rapidly released software tends to have a higher code churn, a higher test coverage and a lower average complexity; challenges in rapid releases are related to managing dependencies and certain code aspects, e.g., design debt.

Background: During the period of one year, ING developed an approach for software analytics within an environment of a large number of software engineering teams working in a Continuous Delivery as a Service setting. Goal: Our objective is to examine what factors helped and hindered the implementation of software analytics in such an environment, in order to improve future software analytics activities. Method: We analyzed artifacts delivered by the software analytics project, and performed semi-structured interviews with 15 stakeholders. Results: We identified 16 factors that helped the implementation of software analytics, and 20 factors that hindered the project. Conclusions: Upfront defining and communicating the aims, standardization of data at an early stage, build efficient visualizations, and an empirical approach help companies to improve software analytics projects.

Context The research literature on software development projects usually assumes that effort is a good proxy for cost. Practice, however, suggests that there are circumstances in which costs and effort should be distinguished. Objectives: We determine similarities and differences between size, effort, cost, duration, and number of defects of software projects. Method: We compare two established repositories (ISBSG and EBSPM) comprising almost 700 projects from industry. Results: We demonstrate a (log)-linear relation between cost on the one hand, and size, duration and number of defects on the other. This justifies conducting linear regression for cost. We establish that ISBSG is substantially different from EBSPM, in terms of cost (cheaper) and duration (faster), and the relation between cost and effort. We show that while in ISBSG effort is the most important cost factor, this is not the case in other repositories, such as EBSPM in which size is the dominant factor. Conclusion: Practitioners and researchers alike should be cautious when drawing conclusions from a single repository.

ING Bank, a large Netherlands-based internationally operating bank, implemented a fully automated continuous delivery pipeline for its software engineering activities in more than 300 teams, that perform more than 2500 deployments to production each month on more than 750 different applications. Our objective is to examine how strong metrics for agile (Scrum) DevOps teams can be set in an iterative fashion. We perform an exploratory case study that focuses on the classification based on predictive power of software metrics, in which we analyze log data derived from two initial sources within this pipeline. We analyzed a subset of 16 metrics from 59 squads. We identified two lagging metrics and assessed four leading metrics to be strong.

Context: In this paper we describe and evaluate a tool for Evidence-Based Software Portfolio Management (EBSPM) that we developed over time in close cooperation with software practitioners from The Netherlands and Belgium. Objectives: The goal of the EBSPM-tool is to measure, analyze, and benchmark the performance of interconnected sets of software projects in terms of size, cost, duration, and number of defects, in order to support innovation of a company's software delivery capability. The tool supports building and maintaining a research repository of finalized software projects from different companies, business domains, and delivery approaches. Method: The tool consists of two parts. First, a Research Repository, at this moment holding data of for now 490 finalized software projects, from four different companies. Second, a Performance Dashboard, built from a so-called Cost Duration Matrix. Results: We evaluated the tool by describing its use in two practical applications in case studies in industry. Conclusions: We show that the EBSPM-tool can be used successfully in an industrial context, especially regarding its benchmarking and visualization purposes.

Context: In this paper we present an exploratory study on the insights of organizations into the perceived value of their software projects. Our study is based on the notion that quantifying and qualifying project size, cost, duration and defects needs to be done in relation with stakeholder satisfaction and perceived value. Objectives: We expect that bringing perceived value into the equation will help in increasing the impact such organizations deliver. Method: In order to find out whether our approach is practically feasible in an industrial setting, we performed an exploratory study in a Belgian telecom company. Results: In this study we evaluate 22 software projects that were delivered during one release. Fifty-three (53) key stakeholders provide stakeholder satisfaction and perceived value measurements in 103 completed surveys. Conclusions: We conclude that a focus on shortening overall project duration, and improving communication on intermediate progress improved stakeholder satisfaction and perceived value. Our study does not provide any evidence that steering on costs helped to improve these.

In this paper we explore opportunities, challenges, and obstacles that Functional Size Measurement (FSM) experts assume to be in automatically derived functional size, directly from the software project code itself. We designed a structured survey, that was answered by 336 FSM specialists. A majority of the respondents consider FSM to be an important tool for decision making. No indications are found for any perceived impact of agile methodology on the difficulty of applying FSM. Respondents overall think of automated FSM as important, but also difficult to realize. 54% of the respondents think that automated FSM will help measurement specialists, while 44% thinks that it will help decision makers too. The most preferred FSM method for automation is COSMIC (25%), followed by IFPUG (21%) and Nesma (16%). Respondents perceive automated FSM to be most suitable for baselining, benchmarking, and maintenance and legacy purposes.

In this paper, we attempt to understand what contributes to a successful process for managing legacy system evolution. We provide an analysis of a number of key performance indicators such as cost, duration, and defects. By normalizing through function points, we furthermore compare to a larger benchmark. To do so we performed a mixed, retrospective case study on a series of nine software releases and eight single once-only releases, all performing on a single, legacy software system, in a West-European telecom company. We interviewed eleven stakeholders that were closely involved in the subject software releases. As a result, we listed a number of observations from the quantitative and qualitative analysis. We found that a release process that performs above average on cost and duration satisfies stakeholders through fast response and direct value, even when the reliability and availability of the actual system is weak.

A medium-sized west-European telecom company experienced a worsening trend in performance, indicating that the organization did not learn from history, in combination with much time and energy spent on preparation and review of project proposals. In order to create more transparency in the supplier proposal process a pilot was started on Functional Size Measurement pricing (FSM-pricing). In this paper we evaluate the implementation of FSM-pricing in the software engineering domain of the company, as an instrument useful in the context of software management and supplier proposal pricing. We analyzed 77 finalized software engineering projects, covering 14 million Euro project cost and a project portfolio size of more than 5,000 function points. We found that a statistical, evidence-based pricing approach for software engineering, as a single instrument (without a connection with expert judgment), can be used in the subject companies to create cost transparency and performance management of software project portfolios.

In this research we aimed to identify distinguishing factors in software releases. For this purpose we analyzed the metrics of 26 software projects. These projects were release-based deliveries from two stable, experienced development teams. During the measurement period both teams transformed from a plan-driven delivery model (waterfall) to an agile approach (Scrum). Overall, we observed that these small release-based projects differ largely from non-release-based projects. Our research indicates that a combination of release-based working, a fixed and experienced development team, and a steady heartbeat contribute to performances that can be characterized as best-practice. The main contribution of this paper is that we found five success factors (all reducing development complexity) that result in best-of-class performance for small software releases.