Refactoring is a widely adopted practice that keeps code healthy and provides well known benefits like improving developer productivity. Developers routinely make decisions about how to refactor code (which specific refactoring changes to make), but the criteria that guide these decisions is not well studied. We conducted a multi-method study to understand the diversity of criteria that developers use in deciding what refactoring changes to make, the relative importance of different criteria, and the extent to which refactoring recommendation tools incorporate these criteria in their recommendation approaches. Our findings demonstrate that developers in industry situationally employ more than a dozen criteria when making refactoring decisions. However, no recommendation tool supports even half of those criteria and most criteria are supported by only a few tools. While research in refactoring recommendations tools is ripe, lack of support for criteria developers care about leaves industry without the kind of recommendation tools that they need. In this paper, we summarize findings from industry interviews, an industry survey, and an analysis of refactoring recommendation tools. We highlight gaps in refactoring recommendation tools that researchers and tool vendors should consider focusing on for successful practical application of refactoring recommendation tools at scale.

Refactoring is a program transformation to improve the internal structure of a program while preserving its external behavior. Developers frequently apply multiple refactorings that depend on each other to achieve goals such as improving code reusability. Although manually applying a sequence of dependent refactorings is a common practice, existing refactoring recommendation tools treat refactorings in isolation without revealing the dependencies among them to developers. One reason is that these relationships among refactorings are poorly understood. Current approaches treat refactoring recommendations as a strictly ordered sequence limiting developers’ ability to understand, validate, and apply recommended refactorings. To address this gap, this paper describes a theory for reasoning about collections of refactorings through defining an ordering dependency relation among refactorings and organizing collection of refactorings as a set of refactoring graphs. We propose an algorithm for identifying refactoring dependencies and illustrate these concepts with a tool for visualizing such refactoring dependencies and refactoring graphs. Our validation results demonstrate that 43% of the 1,457,873 recommended refactorings from 9,595 projects that we studied are part of dependent refactoring graphs. Furthermore, refactorings are not only commonly involved in dependent relations, but also when applied, dependent refactoring graphs improve all of the quality attribute metrics in our experiments more than individual refactorings.

Software-reliant systems need to evolve over time to meet new requirements and take advantage of new technology. However, all too often the structure of software becomes too complex to allow rapid and cost-effective improvements. This increasing complexity is often also a sign of degrading software architecture, making isolating a portion of software for use in a new context or for clean replacement by an improved version difficult. Isolating entangled software from the rest of the architecture typically relies on manual efforts to refactor code that can take thousands of days of effort, as tools provide only limited support for such activities. In this paper, we describe a search-based algorithm that recommends a series of refactorings that collectively isolate specified software from its tangle of architectural dependencies. This approach generates recommendations that reduce problematic dependencies by more than 87% on codebases as large as 1.2M LOC and has the potential to reduce the effort required for this kind of architecture improvement by two-thirds. In walkthroughs, developers found more than 84% of the recommended refactorings acceptable. Our approach provides a much needed foundation for tool support that addresses challenges commonly encountered when improving the architecture of existing software.

Software refactoring plays an important role in software engineering. Developers often turn to refactoring when they want to restructure software to improve its quality without changing its external behavior. Small-scale (floss) refactoring is common in industry and is often performed by a single developer in short sessions, even though developers do much of this work manually instead of using refactoring tools. However, some refactoring efforts are much larger in scale, requiring entire teams and months or years of effort, and the role of tools in these efforts is not as well studied. In this paper, we report on a survey we conducted with developers to understand large-scale refactoring and its tool support needs. Our results from 107 industry developers demonstrate that projects commonly go through multiple large-scale refactorings, each of which requires considerable effort. Our study finds that developers use several categories of tools to support large-scale refactoring and rely more heavily on general-purpose tools like IDEs than on tools designed specifically to support refactoring. Tool support varies across the different activities, with some particularly challenging activities seeing little use of tools in practice. Furthermore, our analysis suggests significant impact is possible through advances in tool support for comprehension and testing, as well as through support for the needs of business stakeholders.

Database normalization is one of the main principles for designing relational databases, which is the most popular database model, with the objective of improving data and system qualities, such as performance. Refactoring the database for normalization can be costly, if the benefits of the exercise are not justified. Developers often ignore the normalization process due to the time and expertise it requires, introducing technical debt into the system. Technical debt is a metaphor that describes trade-offs between short-term goals and applying optimal design and development practices. We consider database normalization debts are likely to be incurred for tables below the fourth normal form. To manage the debt, we propose a multi-attribute analysis framework that makes a novel use of the Portfolio Theory and the TOPSIS method (Technique for Order of Preference by Similarity to Ideal Solution) to rank the candidate tables for normalization to the fourth normal form. The ranking is based on the tables estimated impact on data quality, performance, maintainability, and cost. The techniques are evaluated using an industrial case study of a database-backed web application for human resource management. The results show that the debt-aware approach can provide an informed justification for the inclusion of critical tables to be normalized, while reducing the effort and cost of normalization.

Within any incremental development paradigm, there exists a tension between the desire to deliver value to the customer early and the desire to reduce cost by avoiding architectural refactoring and rework in subsequent releases. What is lacking is an analytical framework that quantifies opportunities and risks of choosing one or the other of these strategies or a blend of the two. This article demonstrates the use of design structure and domain mapping matrices for analyzing architectural dependencies and proposes an optimization-based decision-making technique to support effective release planning. The optimization models recommend the order in which architectural elements and features should be implemented across different releases so as to: (a) minimize rework cost; (b) maximize early value delivery; or (c) optimize an integrated measure of cost and value. These analytic models can be applied earlier in the life cycle and, hence, provide timely information about the progress and changes that occur at each iteration.

Developing machine learning (ML) systems, just like any other system, requires architecture thinking. However, there are characteristics of ML components that create challenges and unique quality attribute (QA) concerns for software architecture and design activities, such as data-dependent behavior, detecting and responding to drift over time, and timely capture of ground truth to inform retraining. This paper presents four categories of software architecture challenges that need to be addressed to support ML system development, maintenance and evolution: software architecture practices for ML systems, architecture patterns and tactics for ML-important QAs, monitorability as a driving QA, and co-architecting and co-versioning. These challenges were collected from targeted workshops, practitioner interviews, and industry engagements. The goal of our work is to encourage further research in these areas and use the information presented in this paper to guide the development of empirically-validated practices for architecting ML systems.

In this paper, we propose intelligent change operators and integrate them into an evolutionary multi-objective search algorithm to recommend valid refactorings that address conflicting quality objectives such as understandability and effectiveness. The proposed intelligent crossover and mutation operators incorporate refactoring dependencies to avoid creating invalid refactorings or invalidating existing refactorings. Further, the intelligent crossover operator is augmented to create offspring that improve solution quality by exchanging blocks of valid refactorings that improve a solution’s weakest objectives. We used our intelligent change operators to generate refactoring recommendations for four widely used open-source projects. The results show that our intelligent change operators improve the diversity of solutions. Diversity is important in genetic algorithms because crossing over a homogeneous population does not yield new solutions. Given the inherent nature of design trade-offs in software, giving developers choices that reflect these trade-offs is important. Higher diversity makes better use of developers time than lots of incredibly similar solutions. Our intelligent change operators also accelerate solution convergence to a feasible solution that optimizes the trade-off between the conflicting quality objectives. Finally, they reduce the number of invalid refactorings by up to 71.52% compared to existing search-based refactoring approaches, and increase the quality of the solutions. Our approach outperformed the state-of-the-art search-based refactoring approaches and an existing deterministic refactoring tool based on manual validation by developers with an average manual correctness, precision and recall of 0.89, 0.82, and 0.87.

Architectural technical debt (ATD) in software-intensive systems refers to the architecture design decisions which work as expedient in the short term, but later negatively impact system evolvability and maintainability. Over the years numerous approaches have been proposed to detect particular types of ATD at a refined level of granularity via source code analysis. Nevertheless, how to gain an encompassing overview of the ATD present in a software-intensive system is still an open question. In this study, we present a multi-step approach designed to build an ATD index (ATDx), which provides insights into a set of ATD dimensions building upon existing architectural rules by leveraging statistical analysis. The ATDx approach can be adopted by researchers and practitioners alike in order to gain a better understanding of the nature of the ATD present in software-intensive systems, and provides a systematic framework to implement concrete instances of ATDx according to specific project and organizational needs.

Despite progress in providing software engineers with tools that automate an increasing number of development tasks, complex activities like redesigning and reengineering existing software remain resource intensive or are supported by tools that are error prone. Complex, but common tasks in industry, like evolving large codebases (1M+ SLOC) to meet changing needs, still rely on costly manual efforts and incur significant technical risk. In one example, an organization that we work with estimated 14,000 hours of development work alone (excluding integration and testing) to isolate a feature from the underlying hardware platform. These examples are pervasive in industry. Software engineering research has taken providing effective tools for software evolution for granted for far too long. The time is right for research to take advantage of advances in search-based software engineering and create the next generation of industry-relevant automated software evolution tools. This paper lays out a vision for automated refactoring at scale towards this goal.

Design problems, frequently the result of optimizing for delivery speed, are a critical part of long-term software costs. Automatically detecting such design issues is a high priority for software practitioners. Software quality tools promise automatic detection of common software quality rule violations. However, since these tools bundle a number of rules, including rules for code quality, it is hard for users to understand which rules identify design issues in particular. Research has focused on comparing these tools on open source projects, but these comparisons have not looked at whether the rules were relevant to design. We conducted an empirical study using a structured categorization approach, and manually classified 466 software quality rules from three industry tools-CAST, SonarQube, and NDepend. We found that most of these rules were easily labeled as either non-design (55%) or design (19%). The remainder (26%) resulted in disagreements among the labelers. Our results are a first step in formalizing a definition of a design rule, to support automatic detection.

Concretely communicating technical debt and its consequences is of common interest to both researchers and software engineers. In the absence of validated tools and techniques to achieve this goal with repeatable results, developers resort to ad hoc practices. Most commonly they report using issue trackers or their existing backlog management practices to capture and track technical debt. In a manual examination of 1,264 issues from four issue trackers from open source industry and government projects, we identified 109 examples of technical debt. Our study reveals that technical debt and its related concepts have entered the vernacular of developers as they discuss development tasks through issue trackers. Even when issues are not explicitly tagged as technical debt, it is possible to identify technical debt items in these issue trackers using a categorization method we developed. We use our results and data to motivate an improved definition and an approach to explicitly report technical debt in issue trackers.

Research in code and architectural analysis has demonstrated that a clear understanding of structural dependencies among software elements helps developers comprehend the impact of change. Yet examples are abundant from industry of major issues due to missed dependencies associated with different views of the architecture. Key concerns include dependencies related to allocation of modules to implementation packages to improve safety-critical testing and allocation of implementation packages to hardware partitions to optimize performance. In this paper, we present an in-depth study of a safety-critical system that underwent major changes as a result of missed architectural dependencies. We describe the challenges that resulted in re-architecting the system, the techniques we used for intervention, our results, and the developers' perspective. While the engineering tools provided coverage of design concerns, they missed implications of end-to-end integration testing, latency, and cost of change. In our study, we observed that the tools led the engineers to focus on data and control flow and therefore to miss many data-entity relationships, resource behavior, and deployment-related dependencies. Research continues to focus on more tooling and automation to assist with dependency analysis rather than interim, easier-to-adopt solutions. Our findings demonstrate that providing developers with a lightweight, semantically well-defined description of dependencies enables them to reason about change impact and propagation implications that they might otherwise overlook.

Software engineers and architects of complex software systems need to balance hard quality attribute requirements while at the same time manage risks and make decisions with a system-wide and long-lasting impact. To achieve these tasks efficiently, they need quantitative information about design-time and run-time system aspects through usable and quick tools. While there is body of work focusing on code quality and metrics, their applicability at the design and architecture level and at scale are inconsistent and not proven. We are interested in exploring whether architecture can assist with better contextualizing existing system and code quality and metrics approaches. Furthermore, we ask whether we need additional architecture-level metrics to make progress and whether something as complex and subtle as software architecture can be quantified. The goal of this workshop is to discuss progress, gather empirical evidence, and identify priorities for a research agenda on architecture and metrics in the software engineering field.

The technical debt metaphor is widely used to encapsulate numerous software quality problems. The metaphor is attractive to practitioners as it communicates to both technical and nontechnical audiences that if quality problems are not addressed, things may get worse. However, it is unclear whether there are practices that move this metaphor beyond a mere communication mechanism. Existing studies of technical debt have largely focused on code metrics and small surveys of developers. In this paper, we report on our survey of 1,831 participants, primarily software engineers and architects working in long-lived, software-intensive projects from three large organizations, and follow-up interviews of seven software engineers. We analyzed our data using both nonparametric statistics and qualitative text analysis. We found that architectural decisions are the most important source of technical debt. Furthermore, while respondents believe the metaphor is itself important for communication, existing tools are not currently helpful in managing the details. We use our results to motivate a technical debt timeline to focus management and tooling approaches.

As industry continues to embrace incremental software development, many projects run into the challenge of incrementally evolving cross-cutting concerns such as performance. To better understand how projects are handling this challenge in practice, we captured experiences from two financial services that made a series of performance improvements over several months. We discovered some commonality in how these projects refine the work, enabling incremental requirements analysis and allocation of work. In this paper, we describe two key aspects of this evolution: refining the concern by breaking it into its constituent parts to drive design tasks and allocating the parts to iterations as the software evolves. Two practices we observed that support this evolution include ratcheting broadened to conceptually describe the refinement approach in dimensions of response to stimuli in a given context and analysis conducted concurrently and loosely coupled from implementation work. This refinement supports ongoing exploration of the problem and solution, and evolutionary development, such as course changes, when new information is acquired.

The goal of this fifth workshop on managing technical debt is to engage leading empiricists and practitioners in exploring practical problems to provide opportunities for research that can provide evidence for or against the emerging definition of technical debt and the efficacy of emerging practice.

Agile projects are showing greater promise in rapid fielding as compared to waterfall projects. However, there is a lack of clarity regarding what really constitutes and contributes to success. We interviewed project teams with incremental development lifecycles, from five government and commercial organizations, to gain a better understanding of success and failure factors for rapid fielding on their projects. A key area we explored involves how Agile projects deal with the pressure to rapidly deliver high-value capability, while maintaining project speed (delivering functionality to the users quickly) and product stability (providing reliable and flexible product architecture). For example, due to schedule pressure we often see a pattern of high initial velocity for weeks or months, followed by a slowing of velocity due to stability issues. Business stakeholders find this to be disruptive as the rate of capability delivery slows while the team addresses stability problems. We found that experienced practitioners, when faced with these challenges, do not apply Agile practices alone. Instead they combine practices - Agile, architecture, or other - in creative ways to respond quickly to unanticipated stability problems. In this paper, we summarize the practices practitioners we interviewed from Agile projects found most valuable and provide an overarching scenario that provides insight into how and why these practices emerge.

The term “Technical Debt” was coined over 20 years ago by Ward Cunningham in a 1992 OOPSL A experience report to de scribe the tr ade-offs between delivering the most appropriate — albeit likely immature — product, in the shortest time possible. Since then the repercussions of going into “technical debt” have become more visible, yet not necessarily more broadly understood. This panel will bring together practitioners to discuss and debate strategies for debt relief.

Although now 20 years old, only recently has the concept of technical debt gained some momentum and credibility in the software engineering community. The goal of this fourth workshop on managing technical debt is to engage researchers and practitioners in exchanging ideas on viable research directions and on how to put the concept to actual use, beyond its usage as a rhetorical instrument to discuss the fate and ailments of software development projects. The workshop participants presented and discussed approaches to detect, analyze, visualize, and manage technical debt, in its various forms, on large software-intensive system developments.

Tools available for measuring the modifiability or impact of change of a system through its architecture typically use structural metrics. These metrics take into account dependencies among the different elements of a system. However, they fail to capture the semantics of an architectural transformation necessary to control the complexity and cost of making changes. To highlight such limitations, this paper presents a study where we applied a representative structural metric, called 'propagation cost', to archetypical architectural transformations known to affect system modifiability such as rearchitecting a tightly coupled system to a layered pattern. We observe that in its original form the propagation cost metric does not provide consistent indications of architecture health. Enhancing this metric based on the semantics of the architectural pattern and tactics used in the transformation show improvements. Our results demonstrate that these enhancements detect modifiability properties that are not detectable by the propagation cost metric.

Summary form only given. This presentation will cover recent developments in the theory of negative imaginary systems and their application to the control of highly resonant flexible structures. The theory of negative imaginary systems arose out of a desire to unify a number of classical methods for the control of lightly damped structures with collocated force actuators and position sensors including positive position feedback and integral force feedback. The key result is a stability result which shows why these methods are guaranteed to yield robust closed loop stability in the face of unmodelled spillover dynamics. Related results to be presented connect the theory of negative imaginary systems to positive real systems theory and a negative imaginary lemma has been established which is analogous to the positive real lemma. The presentation will also discuss recent controller synthesis results based on the theory of negative imaginary systems along with applications in the aerospace area such as the control of large space flexible structures.

The technical debt metaphor is gaining significant traction in the software development community as a way to understand and communicate issues of intrinsic quality, value, and cost. The idea is that developers sometimes accept compromises in a system in one dimension (e.g., modularity) to meet an urgent demand in some other dimension (e.g., a deadline), and that such compromises incur a "debt": on which "interest" has to be paid and which should be repaid at some point for the long-term health of the project. Little is known about technical debt, beyond feelings and opinions. The software engineering research community has an opportunity to study this phenomenon and improve the way it is handled. We can offer software engineers a foundation for managing such trade-offs based on models of their economic impacts. The goal of this second workshop is to discuss managing technical debt as a part of the research agenda for the software engineering field.

Within any incremental development paradigm, there exists a tension between the desire to deliver value to the customer early and the desire to reduce cost by avoiding architectural refactoring in subsequent releases. What is lacking, however, is quantifiable guidance that highlights the potential benefits and risks of choosing one or the other of these alternatives or a blend of both strategies. In this paper, we assert that the ability to quantify architecture quality with measurable criteria provides engineering guidance for iterative release planning. We demonstrate the use of propagation cost as a proxy for architectural health with dependency analysis of design structure and domain mapping matrices as a quantifiable basis for iteration planning.

In any IT-intensive organization, it is useful to have a model to associate a value with software and system architecture decisions. More generally, any effort---a project undertaken by a team---needs to have an associated value to offset its labor and capital costs. Unfortunately, it is extremely difficult to precisely evaluate the benefit of "architecture projects"---those that aim to improve one or more quality attributes of a system via a structural transformation without (generally) changing its behavior. We often resort to anecdotal and informal "hand-waving" arguments of risk reduction or increased developer productivity. These arguments are typically unsatisfying to the management of organizations accustomed to decision-making based on concrete metrics. This paper will discuss research done to address this long-standing dilemma. Specifically, we will present a model derived from analyzing actual projects undertaken at Vistaprint Corporation. The model presented is derived from an analysis of effort tracked against modifications to specific software components before and after a significant architectural transformation to the subsystem housing those components. In this paper, we will discuss the development, implementation, and iteration of the model and the results that we have obtained.

Architecture evolution, a key aspect of software evolution, is typically done in an ad hoc manner, guided only by the competence of the architect performing it. This process lacks the rigor of an engineering discipline. In this paper, we argue that architecture evolution must be engineered - based on rational decisions that are supported by formal models and objective analyses. We believe that evolutions of a restricted form - close-ended evolution, where the starting and ending design points are known a priori - are amenable to being engineered. We discuss some of the key challenges in engineering close-ended evolution. We present a conceptual framework in which an architecture evolutionary trajectory is modeled as a sequence of steps, each captured by an operator. The goal of our framework is to support exploration and objective evaluation of different evolutionary trajectories. We conclude with open research questions in developing this framework.

Software has jumped "out of the box" - it controls critical systems; it pervades business and commerce; it is embedded in myriad mechanisms; it infuses entertainment, communication, and other activities of everyday life. Designs for these applications are constrained not only by traditional considerations of capability and performance but also by economic, business, market, and policy issues and the context of intended use. The diversity of applications requires adaptability in responding to client needs, and the diversity of clients and contexts requires the ability to discriminate among criteria for success.As a result, software designers must also get out of their boxes: in addition to mastering traditional software development skills, they must understand the contextual issues that discriminate good solutions from merely competent ones. Current software engineering education, however, remains largely "in the box": it neglects the rich fabric of issues that lie between the client's problem and actual software development. At Carnegie Mellon we have addressed this major shortcoming with a course that teaches students to understand both the capabilities required by the client and the constraints imposed by the client's context.This paper presents our view of the engineering character of software engineering, describes the content and organization of our new course, reports on our experience from the first three offerings of our course, and suggests ways to adapt our course for other educational settings.