Machine learning (ML) components are increasingly incorporated into software products for end-users, but developers face challenges in transitioning from ML prototypes to products. Academics have limited access to the source of commercial ML products, hindering research progress to address these challenges. In this study, first and foremost, we contribute a dataset of 262 open-source ML products for end users (not just models), identified among more than half a million ML-related projects on GitHub. Then, we qualitatively and quantitatively analyze 30 open-source ML products to answer six broad research questions about development practices and system architecture. We find that the majority of the ML products in our sample represent more startup-style development than reported in past interview studies. We report 21 findings, including limited involvement of data scientists in many open-source ML products, unusually low modularity between ML and non-ML code, diverse architectural choices on incorporating models into products, and limited prevalence of industry best practices such as model testing, pipeline automation, and monitoring. Additionally, we discuss seven implications of this study on research, development, and education, including the need for tools to assist teams without data scientists, education opportunities, and open-source-specific research for privacy-preserving telemetry.

Machine learning models make mistakes, yet sometimes it is difficult to identify the systematic problems behind the mistakes. Practitioners engage in various activities, including error analysis, testing, auditing, and red-teaming, to form hypotheses of what can go (or has gone) wrong with their models. To validate these hypotheses, practitioners employ data slicing to identify relevant examples. However, traditional data slicing is limited by available features and programmatic slicing functions. In this work, we propose SemSlicer, a framework that supports semantic data slicing, which identifies a semantically coherent slice, without the need for existing features. SemSlicer uses Large Language Models to annotate datasets and generate slices from any user-defined slicing criteria. We show that SemSlicer generates accurate slices with low cost, allows flexible trade-offs between different design dimensions, reliably identifies under-performing data slices, and helps practitioners identify useful data slices that reflect systematic problems.

Incorporating machine learning (ML) components into software products raises new software-engineering challenges and exacerbates existing ones. Many researchers have invested significant effort in understanding the challenges of industry practitioners working on building products with ML components, through interviews and surveys with practitioners. With the intention to aggregate and present their collective findings, we conduct a meta-summary study: We collect 50 relevant papers that together interacted with over 4758 practitioners using guidelines for systematic literature reviews. We then collected, grouped, and organized the over 500 mentions of challenges within those papers. We highlight the most commonly reported challenges and hope this meta-summary will be a useful resource for the research community to prioritize research and education in this field.

Abstract Context Robotic systems are known to perform computation-intensive tasks with limited computational resources and battery life. Such systems might benefit from offloading heavy workloads to the Cloud; however, in some cases, this implies high network traffic that degrades performance and energy consumption. Goal In this study, we aim at evaluating the impact of different computation offloading strategies on performance and energy consumption in the context of autonomous robots. Method We conduct two controlled experiments involving a robotic mission based on the Turtlebot3 robot and ROS 1. The mission consists of three tasks that are recurrent in robotics and good candidates for computation offloading in research, namely, SLAM mapping, navigation stack, and object recognition. Each of the tasks is either executed on board or offloaded in a full-factorial experiment design. The obtained measures are then statistically analyzed. Results The results show that offloading the object recognition task causes a more significant decrease in resource utilization and energy consumption than both SLAM mapping and navigation. However, object recognition affects the volume of network traffic significantly to the extent that it can easily cause network congestion. Conclusions In the context of our experiments (i.e.,those involving small-scale ground ROS-based mobile robots operating under WiFi networks), offloading object recognition is beneficial in terms of performance and energy consumption. Nevertheless, large network bandwidth needs to be available for object recognition offloading. While the image resolution and frame rate have a significant impact on not only the network traffic but also energy consumption and performance, these parameters need to be carefully set so that the results of this task can be always received in time, which is particularly crucial in real-time systems.

Computing systems form the backbone of many areas in our society, from manufacturing to traffic control, healthcare, and financial systems. When software plays a vital role in the design, construction, and operation, these systems are referred to as software-intensive systems. Self-adaptation equips a software-intensive system with a feedback loop that either automates tasks that otherwise need to be performed by human operators or deals with uncertain conditions. Such feedback loops have found their way to a variety of practical applications; typical examples are an elastic cloud to adapt computing resources and automated server management to respond quickly to business needs. To gain insight into the motivations for applying self-adaptation in practice, the problems solved using self-adaptation and how these problems are solved, and the difficulties and risks that industry faces in adopting self-adaptation, we performed a large-scale survey. We received 184 valid responses from practitioners spread over 21 countries. Based on the analysis of the survey data, we provide an empirically grounded overview the of state of the practice in the application of self-adaptation. From that, we derive insights for researchers to check their current research with industrial needs, and for practitioners to compare their current practice in applying self-adaptation. These insights also provide opportunities for applying self-adaptation in practice and pave the way for future industry-research collaborations.

The introduction of machine learning (ML) components in software projects has created the need for software engineers to collaborate with data scientists and other specialists. While collaboration can always be challenging, ML introduces additional challenges with its exploratory model development process, additional skills and knowledge needed, difficulties testing ML systems, need for continuous evolution and monitoring, and non-traditional quality requirements such as fairness and explainability. Through interviews with 45 practitioners from 28 organizations, we identified key collaboration challenges that teams face when building and deploying ML systems into production. We report on common collaboration points in the development of production ML systems for requirements, data, and integration, as well as corresponding team patterns and challenges. We find that most of these challenges center around communication, documentation, engineering, and process, and collect recommendations to address these challenges.

Data science pipelines to train and evaluate models with machine learning may contain bugs just like any other code. Leakage between training and test data can lead to overestimating the model’s accuracy during offline evaluations, possibly leading to deployment of low-quality models in production. Such leakage can happen easily by mistake or by following poor practices, but may be tedious and challenging to detect manually. We develop a static analysis approach to detect common forms of data leakage in data science code. Our evaluation shows that our analysis accurately detects data leakage and that such leakage is pervasive among over 100,000 analyzed public notebooks. We discuss how our static analysis approach can help both practitioners and educators, and how leakage prevention can be designed into the development process.

In software engineering, energy awareness refers to the conscious design and development of software that is able to monitor and react to energy state. Energy awareness is the key building block for energy efficiency and for other quality aspects of robotics software, such as mission completion time and safety. However, as of today, there is no guidance for practitioners and researchers on how to architect robotics software with energy awareness in mind. The goal of this paper is to identify architectural tactics for energy-aware robotics software. Specifically, using a dataset of 339493 data points extracted from five complementary data sources (e.g., source code repositories, Stack Overflow), we identified and analyzed 97 data points that considered both energy consumption and architectural concerns. We then synthesized a set of energy-aware architectural tactics via thematic analysis. In this preliminary investigation we focus on two representative architectural tactics.

Developing machine learning (ML) systems, just like any other system, requires architecture thinking. However, there are characteristics of ML components that create challenges and unique quality attribute (QA) concerns for software architecture and design activities, such as data-dependent behavior, detecting and responding to drift over time, and timely capture of ground truth to inform retraining. This paper presents four categories of software architecture challenges that need to be addressed to support ML system development, maintenance and evolution: software architecture practices for ML systems, architecture patterns and tactics for ML-important QAs, monitorability as a driving QA, and co-architecting and co-versioning. These challenges were collected from targeted workshops, practitioner interviews, and industry engagements. The goal of our work is to encourage further research in these areas and use the information presented in this paper to guide the development of empirically-validated practices for architecting ML systems.

In today's world, reducing energy consumption should be the goal for any organization and any system, including robotics software systems. However, state of the practice in robotics software development focuses primarily on achieving functionality and performance, with minimal recognition of energy-efficiency as a driving software quality. The goal of this paper is to identify, synthesize, and empirically evaluate architectural tactics for energy-efficiency applied by practitioners in real robotics projects. Four tactics were identified by mining software repository techniques applied to the ROS ecosystem. The tactics were evaluated via experimentation on a real, commodity robotics system. Results show that the application of green architectural tactics tends to largely improve the energy-efficiency of the robot (7.9% energy savings when all tactics are applied) and that the movement strategy and the physical environment where the robot operates strongly influence how energy is consumed by the robot.

The Robot Operating System (ROS) is the de-facto standard for robotics software. However, ROS-based systems are getting larger and more complex and could benefit from good software architecture practices. We aim at (i) unveiling the state-of-the-practice in terms of targeted quality attributes and architecture documentation in ROS-based systems, and (ii) providing empirically-grounded guidance to roboticists about how to properly architect ROS-based systems. We designed and conducted an observational study where we (i) built a dataset of 335 GitHub repositories containing real open-source ROS-based systems, and (ii) mined the repositories to extract and synthesize quantitative and qualitative findings about how roboticists are architecting ROS-based systems. First, we extracted an empirically-grounded overview of the state of the practice for architecting and documenting ROS-based systems. Second, we synthesized a catalog of 47 architecting guidelines for ROS-based systems. Third, the extracted guidelines were validated by 119 roboticists working on real-world open-source ROS-based systems. Roboticists can use our architecting guidelines for applying good design principles to develop robots that meet quality requirements, and researchers can use our results as evidence-based indications about how real-world ROS systems are architected today, thus inspiring future research contributions.

Since its founding in 1989, the Studio-based Master of Software Engineering (MSE) Program at Carnegie Mellon University (CMU) has been a trailblazer in advancing the practice of masters-level professional software engineering education. From its inception, the program was designed around a comprehensive development project, the Studio. The Studio provides students with a team-based, mentored, multi-semester engagement with external clients, allowing them to "learn by doing" as they apply skills and techniques derived from complementary core and elective courses. Going far beyond the common "capstone" experience of many other programs at the time, the Studio concept was unique because of the amount of time set aside for practice-based learning (almost a third of the entire curriculum), its approach to active mentoring (engaging seasoned software engineering professionals), and its focus on reflective practices (with explicit attention given to retrospective analysis of positive and negative experiences). This formulation of a software engineering professional degree program has had far-reaching and lasting impacts on software engineering education.

The objective of the panel is to discuss how to enhance diversity in software engineering programs. Brief possition statements from the panelists are provided. A complete record of the panel discussion was not made available for publication as part of the conference proceedings.

The context of a software system comprises the knowledge that architects need to have about the environment in which a system is expected to operate. Contextual knowledge, however, is often unknown or overlooked. This results in software architects designing systems based on assumptions that are largely unfounded and can potentially lead to system failures. To address this problem, this paper presents a Context Description Viewpoint that captures context in software architecture. The viewpoint is based on the results of a literature review that analyzed the state-of-the-art in context, its elements, and modeling techniques. We evaluated and revised the viewpoint by using two case studies based on real-world projects. The case studies showed that the viewpoint is expressive enough to capture context. For software architects it represents a reusable work product to design software systems and to help them identify, capture, and analyze contextual knowledge.

Cyber-foraging is a technique to enable mobile devices to extend their computing power and storage by offloading computation or data to more powerful servers located in the cloud or in single-hop proximity. While there is a large amount of research in this area, the reality is that there are not many deployed, operational cyber-foraging systems. As these systems become more prevalent due to their proven benefits, combined with the emergence of micro data centers and edge clouds, a need will arise for guidance on their architecture and development. To provide this guidance, we present a decision model based on a mapping of functional and non-functional requirements for cyber-foraging systems to a set of architectural tactics. The decision model was validated by developers to obtain an expert opinion on its correctness and usefulness for guiding the architecture, design, and evolution of cyber-foraging systems that meet their intended functional and non-functional requirements, while understanding the effects of decisions.

Energy efficiency is a primary concern for the ICT sector. In particular, the widespread adoption of cloud computing technologies has drawn attention to the massive energy consumption of data centers. Although hardware constantly improves with respect to energy efficiency, this should also be a main concern for software. In previous work we analyzed the literature and elicited a set of techniques for addressing energy efficiency in cloud-based software architectures. In this work we codified these techniques in the form of Green Architectural Tactics. These tactics will help architects extend their design reasoning towards energy efficiency and to apply reusable solutions for greener software.

The convergence of mobile computing and cloud computing enables new multimedia applications that are both resource-intensive and interaction-intensive. For these applications, end-to-end network bandwidth and latency matter greatly when cloud resources are used to augment the computational power and battery life of a mobile device. We first present quantitative evidence that this crucial design consideration to meet interactive performance criteria limits data center consolidation. We then describe an architectural solution that is a seamless extension of today's cloud computing infrastructure.

PESOS 2013 is a forum that brings together software engineering researchers from academia and industry, as well as practitioners working in the areas of service-oriented systems to discuss research challenges, recent developments, novel application scenarios, as well as methods, techniques, experiences, and tools to support engineering, evolution and adaptation of service-oriented systems. The special theme of the 5th edition of PESOS is “Service Engineering for the Cloud” The goal is to explore approaches to better engineer service-oriented systems, to either take advantage of the qualities offered by cloud infrastructures or to account for lack of full control over important quality attributes. PESOS 2013 also continues to be the key forum for collecting case studies and artifacts for educators and researchers in this area.

Mobile-enabled systems make use of mobile devices, RFID tags, sensor nodes, and other computing-enabled mobile devices to gather contextual data from users and the surrounding changing environment. Such systems produce computational data that can be stored and used in the field, shared between mobile and resident devices, and potentially uploaded to local servers or the cloud — a distributed, heterogeneous, context-aware, data production and consumption paradigm. Mobile-enabled systems have characteristics that make them different from traditional systems, such as limited resources, increased vulnerability, performance and reliability variability, and a finite energy source. There is significantly higher unpredictability in the execution environment of mobile apps. This workshop brings together experts from the software engineering and mobile computing communities — with notable participation from researchers and practitioners in the field of distributed systems, enterprise systems, cloud systems, ubiquitous computing, wireless sensor networks, and pervasive computing — to share results and open issues in the area of software engineering of mobile-enabled systems.

Structure makes data more useful, but also makes data entry more cumbersome. Studies have found that this is especially true on mobile devices, as mobile users often reject structured personal information management tools because the structure is too restrictive and makes entering data slower. To overcome these problems, we introduce a new data entry technique that lets users create customized structured data in an unstructured manner. We use a novel notepad-like editing interface with built-in data detectors that allow users to specify structured data implicitly and reuse the structures when desired. To minimize the amount of typing, it provides intelligent, context-sensitive autocomplete suggestions using personal and public databases that contain candidate information to be entered. We implemented these mechanisms in an example application called Listpad. Our evaluation shows that people using Listpad create customized structured data 16% faster than using a conventional mobile database tool. The speed further increases to 42% when the fields can be autocompleted.

A key use for mobile devices is to search and view online information while on the go. As a result, many mobile applications serve as front ends for online databases. While there are many thousands of data sources that provide web service APIs giving access to their databases, creating mobile applications to use those sources requires significant mobile programming knowledge and a significant amount of time. We introduce Spinel, a plug-in architecture for Android, and a set of web-based configuration tools that together enable users to connect mobile applications to new data sources without programming. Spinel also provides APIs that make it easy for developers to create new applications that use those data sources. We provide three demonstration Android applications that use such data: Listpad for entering personal lists, Listviewer for viewing results of data queries, and Mapviewer for displaying query results on a map. An informal usability study showed that users could successfully attach new data sources to those applications.

Service-oriented systems have attracted great interest from industry and research communities worldwide. Service integrators, developers, and providers are collaborating to address the various challenges in the field. PESOS 2011 is a forum for all these communities to present and discuss a wide range of topics related to service-oriented systems. The goal of PESOS is to bring together researchers from academia and industry, as well as practitioners working in the areas of software engineering and service-oriented systems to discuss research challenges, recent developments, novel applications, as well as methods, techniques, experiences, and tools to support the engineering of service-oriented systems.

Service-oriented systems have attracted great interest from industry and research communities worldwide. Service integrators, developers, and providers are collaborating to address the various challenges in the field. PESOS 2010 is a forum for all these communities to present and discuss a wide range of topics related to service-oriented systems. The goal of PESOS was to bring together researchers from academia and industry, as well as practitioners working in the areas of software engineering and service-oriented systems to discuss research challenges, recent developments, novel applications, as well as methods, techniques, experiences, and tools to support the engineering and use of service-oriented systems.

The main goal of this workshop is to provide an opportunity for participants to present current work and have a lively discussion of open issues for the maintenance and evolution of service-oriented systems. The dialog will include both short-term research issues such as reengineering processes and long-term issues such as evolution patterns for service-oriented systems.

Service-Oriented Architecture (SOA) is a way of designing, developing, deploying, and managing enterprise systems where business needs and technical solutions are closely aligned. SOA offers a number of potential benefits, such as cost-efficiency and agility. However, adopting SOA is not without considerable challenges. For example, the most common way to implement a SOA-based system is with Web services, but the standards that define Web services are evolving rapidly and many of the Web services tools are still somewhat immature. There is also the question of how to leverage existing legacy assets within a SOA context. Perhaps most importantly, there are serious challenges related to the testing of SOA-based systems that must be addressed before the SOA paradigm will enjoy broadbased success.

The main goal of this workshop is to provide an opportunity for participants to present current work and have a lively discussion of open issues for the maintenance and evolution of service-oriented systems. The dialog will include both short-term research issues such as reengineering processes and long-term issues such as evolution patterns for service-oriented systems. The current version of a SOA research agenda will be used as a starting point for discussion.

The main goal of this working session is to provide an opportunity for participants to present current work and have a lively discussion of open issues for the maintenance and evolution of SOA-based systems. The dialog will include both business issues (e.g., organizational changes in the context of migrating to an SOA environment) and technical issues (e.g., analyzing the reuse potential of legacy components in an SOA environment). A draft version of a research agenda will be used as a starting point for obtaining feedback from researchers and practitioners active in this area.

Dissatisfaction with heavyweight development approaches has lead to a proposal of agile methodologies. While heavyweight approaches are geared towards careful project planning, formalized quality assurance, and controlled and rigorous software development processes, the agile approaches focus on quick adaptation to change, and flexibility. Today, there are a number of different agile methods available, such as Scrum, Crystal, Adaptive Software Development, Feature Driven Development, and extreme programming. Although different, they share the same principles. Some of them are: (1) individuals and interactions over processes and tools, (2) working code over comprehensive documentation, (3) customer collaboration over contract negotiation, and (4) responding to change over following a plan. The proponents claim that agile methodologies lead to more effective and efficient development. The opponents, on the other hand, have stamped the agile approach as a "license to hack" or "cowboy", or "code-and-fix" method.

Planning and management of software sustainment is impaired by a lack of consistently applied, practical measures. Without these measures, it is impossible to determine the effect of efforts to improve sustainment practices. In this paper we provide a context for evaluating sustainability and discuss a set of measures developed at the Software Engineering Institute at Carnegie Mellon University.

The question of which design methods are appropriate for component-based development (CBD) is complicated by different understandings of the end objectives of CBD. A further complication is different understandings of what is meant by "component." These differences lead to entirely distinct classes of design problem. The aim of this tutorial is to, first, outline the different classes of design problem that span these differing interpretations of CBD, and, second, to outline the required methodological responses to these design problems.