The recent advancements of Small Language Models (SLMs) have opened new possibilities for efficient code generation. SLMs offer lightweight and cost-effective alternatives to Large Language Models (LLMs), making them attractive for use in resource-constrained environments. However, empirical understanding of SLMs, particularly their capabilities, limitations, and performance trade-offs in code generation remains limited. This study presents a comprehensive empirical evaluation of 20 open-source SLMs ranging from 0.4B to 10B parameters on five diverse code-related benchmarks (HumanEval, MBPP, Mercury, HumanEvalPack, and CodeXGLUE). The models are assessed along three dimensions: i) functional correctness of generated code, ii) computational efficiency and iii) performance across multiple programming languages. The findings of this study reveal that several compact SLMs achieve competitive results while maintaining a balance between performance and efficiency, making them viable for deployment in resource-constrained environments. However, achieving further improvements in accuracy requires switching to larger models. These models generally outperform their smaller counterparts, but they require much more computational power. We observe that for 10% performance improvements, models can require nearly a 4x increase in VRAM consumption, highlighting a trade-off between effectiveness and scalability. Besides, the multilingual performance analysis reveals that SLMs tend to perform better in languages such as Python, Java, and PHP, while exhibiting relatively weaker performance in Go, C++, and Ruby. However, statistical analysis suggests these differences are not significant, indicating a generalizability of SLMs across programming languages. Based on the findings, this work provides insights into the design and selection of SLMs for real-world code generation tasks.

The use of Large Language Models (LLMs) for autonomous code generation is gaining attention in emerging technologies. As LLM capabilities expand, they offer new possibilities such as code refactoring, security enhancements, and legacy application upgrades. Many outdated web applications pose security and reliability challenges, yet companies continue using them due to the complexity and cost of upgrades. To address this, we propose an LLM-based multi-agent system that autonomously upgrades legacy web applications to the latest versions. The system distributes tasks across multiple phases, updating all relevant files. To evaluate its effectiveness, we employed Zero-Shot Learning (ZSL) and One-Shot Learning (OSL) prompts, applying identical instructions in both cases. The evaluation involved updating view files and measuring the number and types of errors in the output. For complex tasks, we counted the successfully met requirements. The experiments compared the proposed system with standalone LLM execution, repeated multiple times to account for stochastic behavior. Results indicate that our system maintains context across tasks and agents, improving solution quality over the base model in some cases. This study provides a foundation for future model implementations in legacy code updates. Additionally, findings highlight LLMs' ability to update small outdated files with high precision, even with basic prompts. The source code is publicly available on GitHub: https://github.com/alasalm1/Multi-agent-pipeline.

The growing need for scalable, maintainable, and fast-deploying systems has made microservice architecture widely popular in software development. This paper presents a system that uses Large Language Models (LLMs) to automate the API-first development of RESTful microservices. This system assists in creating OpenAPI specification, generating server code from it, and refining the code through a feedback loop that analyzes execution logs and error messages. By focusing on the API-first methodology, this system ensures that microservices are designed with well-defined interfaces, promoting consistency and reliability across the development life-cycle. The integration of log analysis enables the LLM to detect and address issues efficiently, reducing the number of iterations required to produce functional and robust services. This process automates the generation of microservices and also simplifies the debugging and refinement phases, allowing developers to focus on higher-level design and integration tasks. This system has the potential to benefit software developers, architects, and organizations to speed up software development cycles and reducing manual effort. To assess the potential of the system, we conducted surveys with six industry practitioners. After surveying practitioners, the system demonstrated notable advantages in enhancing development speed, automating repetitive tasks, and simplifying the prototyping process. While experienced developers appreciated its efficiency for specific tasks, some expressed concerns about its limitations in handling advanced customizations and larger-scale projects. The code is publicly available at https://github.com/sirbh/code-gen.

Context: Early-stage software startups, despite their strong innovative potential, experience high failure rates due to factors such as inexperience, limited resources, and market uncertainty. Generative AI technologies, particularly Large Language Models (LLMs), offer promising support opportunities; however, effective strategies for their integration into startup practices remain underexplored. Objective: This study investigates how prompt engineering and system instruction design can enhance the utility of LLMs in addressing the specific needs and challenges faced by early-stage software startups. Methods: A Design Science Research (DSR) methodology was adopted, structured into three iterative cycles. In the first cycle, use cases for LLM adoption within the startup context were identified. The second cycle experimented with various prompt patterns to optimize LLM responses for the defined use cases. The third cycle developed “StartupGPT”, an LLM-based assistant tailored for startups, exploring system instruction designs. The solution was evaluated with 25 startup practitioners through a combination of qualitative feedback and quantitative metrics. Results: The findings show that tailored prompt patterns and system instructions significantly enhance user perceptions of LLM support in real-world startup scenarios. StartupGPT received strong evaluation scores across key dimensions: satisfaction (93.33%), effectiveness (80%), efficiency (80%), and reliability (86.67%). Nonetheless, areas for improvement were identified, particularly in context retention, personalization of suggestions, communication tone, and sourcing external references. Conclusion: This study empirically validates the applicability of LLMs in early-stage software startups. It offers actionable guidelines for prompt and system instruction design and contributes both theoretical insights and a practical artifact — StartupGPT — that supports startup operations without necessitating costly LLM retraining.

Context: Generative AI (GenAI) is argued to transform software engineering (SE) in various ways, and GenAI tools show promise for various SE tasks. Software organizations across the globe are currently exploring the use of GenAI for SE. Objective: While numerous studies have recently been published on GenAI, few studies have looked at the adoption of these tools and their usage from an organizational point of view, focusing instead on individual users. Our objective is to understand how organizations adopt these tools and what their impacts are in industrial contexts, with a focus on the European perspective. Method: We conducted a multiple case study of seven European companies. We collected data through semi-structured interviews (n=15), as well as through longitudinal observation in one case company. All data were analyzed using thematic analysis. Results: We analyzed 28 transcripts, resulting in 456 quotations and 557 code occurrences split between 66 individual codes that were categorized under 6 high-level themes. We identified 25 types of tasks GenAI was currently being used for in our case organizations. We identified 12 benefits for GenAI in SE and 10 adoption and use challenges. Key adoption challenges for organizations include data privacy and legislative concerns, the emerging and fast-moving market of GenAI tools, difficulty of measuring the positive impact of the tools, and potential change resistance. For individuals, the key challenges are related to prompting, such as understanding what a good prompt is, and how to write prompts for specific tasks. Conclusion: GenAI adoption is becoming widespread in SE, but good practices and use cases are still emerging. While GenAI can potentially produce various benefits in SE, companies and individual users are facing various challenges in making the most of GenAI in SE. Our results provide insights into the current state of practice.

ABSTRACT Context Generative artificial intelligence (GenAI) tools have become increasingly prevalent in software development, offering assistance to various managerial and technical project activities. Notable examples of these tools include OpenAI's ChatGPT, GitHub Copilot, and Amazon CodeWhisperer. Objective Although many recent publications have explored and evaluated the application of GenAI, a comprehensive understanding of the current development, applications, limitations, and open challenges remains unclear to many. Particularly, we do not have an overall picture of the current state of GenAI technology in practical software engineering usage scenarios. Method We conducted a literature review and focus groups for a duration of five months to develop a research agenda on GenAI for software engineering. Results We identified 78 open research questions (RQs) in 11 areas of software engineering. Our results show that it is possible to explore the adoption of GenAI in partial automation and support decision‐making in all software development activities. While the current literature is skewed toward software implementation, quality assurance and software maintenance, other areas, such as requirements engineering, software design, and software engineering education, would need further research attention. Common considerations when implementing GenAI include industry‐level assessment, dependability and accuracy, data accessibility, transparency, and sustainability aspects associated with the technology. Conclusions GenAI is bringing significant changes to the field of software engineering. Nevertheless, the state of research on the topic still remains immature. We believe that this research agenda holds significance and practical value for informing both researchers and practitioners about current applications and guiding future research.

[Context]: The COVID-19 pandemic has disrupted the global market and workplace landscape. As a response, hybrid work situations have become popular in the software business sector. This way of working has an impact on software companies. [Objective]: This study investigates software companies' responses to hybrid working. [Method]: We conducted a large-scale survey to achieve our objective. Our results are based on a qualitative analysis of 124 valid responses. [Results]: The main result of our study is a taxonomy of software companies' impacts on hybrid working at individual, team and organisation levels. We found higher positive responses at individual and organisational levels than negative responses. At the team level, both positive and negative impacts obtained a uniform number of responses. [Conclusion]: The results indicate that hybrid working became credible with the wave of COVID-19, with 83 positive responses outweighing the 41 negative responses. Software company respondents witnessed better work-life balance, productivity, and efficiency in hybrid working.

Context The COVID‐19 pandemic has had a disruptive impact on how people work and collaborate across all global economic sectors, including software business. While remote working is not new for software engineers, forced WFH situations come with both limitations and opportunities. As the ‘new normal’ for working might be based on the current state of Work‐from‐home (WFH), it is useful to understand what has happened and learn from that. Objective This study aims to gain insights into how their WFH arrangement impacts project management and software engineering. We are also interested in exploring these impacts in different contexts, such as startups and established companies. Method We conducted a global‐scale, cross‐sectional survey during the spring and summer 2021. Our results are based on quantitative and qualitative analysis of 297 valid responses. Results We characterize the profile of WFH in both spatial and temporal aspects, together with a set of common collaborative tools and coordination and control mechanisms. We revealed some areas of project management that are relatively more challenging during WFH situations, such as coordination, communication and project planning. We also revealed a mixed picture of the perceived impact of WFH on different software engineering activities. Conclusion WFH is a situational phenomenon which can have both negative and positive impact on software teams. For practitioners, we suggest a unified approach to consider the context of WFH, collaborative tools, associated coordination and control approaches and a process that resolve those aspects that are sensitive to physical interaction.

Software startups are important drivers of economy on a global scale, and have become associated with innovation and high growth. However, the overwhelming majority of startups ends in failure. Many of these startup failures ultimately stem from software engineering issues, and requirements engineering (RE) ones in particular. Despite the emphasis placed on the importance of RE activities in the startup context, many startups continue to develop software without a clear market or customer, having never had meaningful contact with their would-be customer. We develop a method aimed at early-stage startups that is intended to help startups through the initial stages of the startup process: StartCards. The method emphasizes the importance of idea and product validation activities in particular in order to tackle anti-patterns related to (a lack of) RE in startups. This method is based on existing literature, both grey and academic literature. StartCards was developed using the Canonical Action Research (CAR) approach, over the course of 4 AR cycles. During the AR process, the method was used by 44 student startup teams in a practical course setting. Data from the use of the method was collected through self-reporting in the form of modified learning diaries, mentoring meetings with the startup teams, and a qualitative survey. We consider the current version of StartCards useful for early-stage startups based on the data we have collected. The method can also be used as a pedagogical tool in startup education. The paper presents the first published version of the method. While work on the method continues, the method is deemed ready for use.

The public and academic discussion on Artificial Intelligence (AI) ethics is accelerating and the general public is becoming more aware AI ethics issues such as data privacy in these systems. To guide ethical development of AI systems, governmental and institutional actors, as well as companies, have drafted various guidelines for ethical AI. Though these guidelines are becoming increasingly common, they have been criticized for a lack of impact on industrial practice. There seems to be a gap between research and practice in the area, though its exact nature remains unknown. In this paper, we present a gap analysis of the current state of the art by comparing practices of 39 companies that work with AI systems to the seven key requirements for trustworthy AI presented in the “The Ethics Guidelines for Trustworthy Artificial Intelligence”. The key finding of this paper is that there is indeed notable gap between AI ethics guidelines and practice. Especially practices considering the novel requirements for software development, requirements of societal and environmental well-being and diversity, nondiscrimination and fairness were not tackled by companies.

Abstract Artificial Intelligence (AI) systems are increasing in significance within software services. Unfortunately, these systems are not flawless. Their faults, failures and other systemic issues have emphasized the urgency for consideration of ethical standards and practices in AI engineering. Despite the growing number of studies in AI ethics, comparatively little attention has been placed on how ethical issues can be mitigated in software engineering (SE) practice. Currently understanding is lacking regarding the provision of useful tools that can help companies transform high-level ethical guidelines for AI ethics into the actual workflow of developers. In this paper, we explore the idea of using user stories to transform abstract ethical requirements into tangible outcomes in Agile software development. We tested this idea by studying master’s level student projects (15 teams) developing web applications for a real industrial client over the course of five iterations. These projects resulted in 250+ user stories that were analyzed for the purposes of this paper. The teams were divided into two groups: half of the teams worked using the ECCOLA method for AI ethics in SE, while the other half, a control group, was used to compare the effectiveness of ECCOLA. Both teams were tasked with writing user stories to formulate customer needs into system requirements. Based on the data, we discuss the effectiveness of ECCOLA, and Primary Empirical Contributions (PECs) from formulating ethical user stories in Agile development.

Artificial Intelligence (AI) systems are becoming increasingly widespread and exert a growing influence on society at large. The growing impact of these systems has also highlighted potential issues that may arise from their utilization, such as data privacy issues, resulting in calls for ethical AI systems. Yet, how to develop ethical AI systems remains an important question in the area. How should the principles and values be converted into requirements for these systems, and what should developers and the organizations developing these systems do? To further bridge this gap in the area, in this paper, we present a method for implementing AI ethics: ECCOLA. Following a cyclical action research approach, ECCOLA has been iteratively developed over the course of multiple years, in collaboration with both researchers and practitioners.

Business Model Canvas (BMC) is a tool widely used to describe startup business models. Despite the various business aspects described, BMC pays a little emphasis on team- related factors. The importance of team-related factors in software development has been acknowledged widely in literature. While not as extensively studied, the importance of teams in software startups is also known in both literature and among practitioners. In this paper, we propose potential changes to BMC to have the tool better reflect the importance of the team, especially in a software startup environment. Based on a literature review, we identify various components related to the team, which we then further support with empirical data. We do so by means of a qualitative case study of five startups.

Startups are often seen as drivers of innovation. In an attempt to leverage this potential, larger business organizations have founded internal startups as a subset of internal corporate ventures (ICV). These smaller organizations are intended to be more agile than the parent organization, in order to produce new service and product innovations using their own methods and practices independently of the organizational culture and methods of the parent organization. However, our understanding of ICVs is still lacking in terms of processes and success factors, and especially the more recent internal startups have scarcely been studied thus far. To approach this novel area of research, we take on a qualitative approach by means of a multiple case study of internal startups in large companies. Based on the data, we argue that the origin of the idea of the internal startup heavily influences the processes utilized by the internal startup, as well as the connections between the internal startup and its parent organization. We also highlight various practical implications.

Various recent Artificial Intelligence (AI) system failures, some of which have made the global headlines, have highlighted issues in these systems. These failures have resulted in calls for more ethical AI systems that better take into account their effects on various stakeholders. However, implementing AI ethics into practice is still an on-going challenge. High-level guidelines for doing so exist, devised by governments and private organizations alike, but lack practicality for developers. To address this issue, in this paper, we present a method for implementing AI ethics. The method, ECCOLA, has been iteratively developed using a cyclical action design research approach. The method aims at making the high-level AI ethics principles more practical, making it possible for developers to more easily implement them in practice.

Artificial Intelligence (AI) solutions are becoming increasingly common in software development endeavors, and consequently exert a growing societal influence as well. Due to their unique nature, AI based systems influence a wide range of stakeholders with or without their consent, and thus the development of these systems necessitates a higher degree of ethical consideration than is currently carried out in most cases. Various practical examples of AI failures have also highlighted this need. However, there is only limited research on methods and tools for implementing AI ethics in software development, and we currently have little knowledge of the state of practice. In this study, we explore the state of the art in startup-like environments where majority of the AI software today gets developed. Based on a multiple case study, we discuss the current state of practice and highlight issues. The cases underline the complete ignorance of ethical consideration in AI endeavors. We also outline existing good practices that can already support the implementation of AI ethics, such as documentation and error handling.

Software Engineers work using highly diverse methods and practices, and general theories in software engineering are lacking. A recent attempt at creating a common ground in the area of software engineering methodologies has been the Essence Theory of Software Engineering. Essence is a method-agnostic progress management framework and a meta-method for Software Engineering (SE). However, tooling for Essence is still lacking. Without dedicated tools and other instruments, a meta-method such as Essence is cumbersome to utilize by practitioners and students. Indeed, Essence currently suffers from a lack of widespread practitioner adoption. In this paper, we thus present an Open Source tool for essentializing methods and practices: Essencery. We conduct a qualitative evaluation of the tool through a quasi-formal experiment and a set of semi-structured interviews. Based on this data, we improve Essencery iteratively before it is utilized in a large-scale project-based course as a proof of concept.

Cyber Physical Systems (CPS) development approaches tend to start from the physical (hardware) perspective, and the software is the final element in the process. However, this approach is unfit for the more software-intensive world that is increasingly iterative, connected, and constantly online. Many constraints prevent the application of iterative, incremental, and agile development methodologies, which now are the norm for many other fields of software. Time-consuming system validation can only start when both hardware and software components are ready, which implies that the software delivery and quality is almost always the final bottleneck in the CPS development and integration. Also organizational issues raise concerns - CPS development teams are nowadays often geographically distributed, which can result in delays in the process, shortcomings, and even mistakes. In this paper, we propose using our envisioned open-source Virtual Reality-based Integrated software Development Environment (VRIDE) for developing the next generation, increasingly software-intensive CPSs in efficient ways.

Use of artificial intelligence (AI) in human contexts calls for ethical considerations for the design and development of AI-based systems. However, little knowledge currently exists on how to provide useful and tangible tools that could help software developers and designers implement ethical considerations into practice. In this paper, we empirically evaluate a method that enables ethically aligned design in a decision-making process. Though this method, titled the RESOLVEDD strategy, originates from the field of business ethics, it is being applied in other fields as well. We tested the RESOLVEDD strategy in a multiple case study of five student projects where the use of ethical tools was given as one of the design requirements. A key finding from the study indicates that simply the presence of an ethical tool has an effect on ethical consideration, creating more responsibility even in instances where the use of the tool is not intrinsically motivated.