Large Language Models (LLMs) are powerful tools used in several applications such as conversational AI, and code generation. However, significant robustness concerns arise with LLMs in production, such as hallucinations, prompt injection attacks, harmful content generation, and challenges in maintaining accurate domain-specific content moderation. Guardrails aim to mitigate these challenges by aligning LLM outputs with desired behaviors without modifying the underlying models. Nvidia NeMo Guardrails, for instance, rely on specifying acceptable/unacceptable behaviours. However, it is challenging to predict and address potential issues of LLMs in advance to create these guardrails. Also, manual updates from software engineers are often required to maintain and refine these guardrails. We introduce LLM-Guards, specialised machine learning (ML) models trained to function as protective guards. Additionally, we present an automation pipeline for training and continual fine-tuning of these guards using reinforcement learning from human feedback (RLHF). We evaluated several small LLMs, including Llama-3, Mistral, and Gemma, as LLM-Guards for challenges such as moderation and detecting off-topic queries, and compared their performance against NeMo Guardrails. The proposed Llama-3 LLM-Guard outperformed NeMo Guardrails in detecting offtopic queries, achieving an accuracy of 98.7% compared to 81%. Furthermore, the LLM-Guard detected 97.86% of harmful queries” surpassing NeMo Guardrails by 19.86%.

Machine learning (ML), especially with the emergence of large language models (LLMs), has significantly transformed various industries. However, the transition from ML model prototyping to production use within software systems presents several challenges. These challenges primarily revolve around ensuring safety, security, and transparency, subsequently influencing the overall robustness and trustworthiness of ML models. In this paper, we introduce ML-On-Rails, a protocol designed to safeguard ML models, establish a well-defined endpoint interface for different ML tasks, and clear communication between ML providers and ML consumers (software engineers). ML-On-Rails enhances the robustness of ML models via incorporating detection capabilities to identify unique challenges specific to production ML. We evaluated the ML-On-Rails protocol through a real-world case study of the MoveReminder application. Through this evaluation, we emphasize the importance of safeguarding ML models in production.

Machine learning (ML) models have become essential components in software systems across several domains, such as autonomous driving, healthcare, and finance. The robustness of these ML models is crucial for maintaining the software systems performance and reliability. A significant challenge arises when these systems encounter out-of-distribution (OOD) data, examples that differ from the training data distribution. OOD data can cause a degradation of the software systems performance. Therefore, an effective OOD detection mechanism is essential for maintaining software system performance and robustness. Such a mechanism should identify and reject OOD inputs and alert software engineers. Current OOD detection methods rely on hyperparameters tuned with in-distribution and OOD data. However, defining the OOD data that the system will encounter in production is often infeasible. Further, the performance of these methods degrades with OOD data that has similar characteristics to the in-distribution data. In this paper, we propose a novel OOD detection method using the Gini coefficient. Our method does not require prior knowledge of OOD data or hyperparameter tuning. On common benchmark datasets, we show that our method outperforms the existing maximum softmax probability (MSP) baseline. For a model trained on the MNIST dataset, we improve the OOD detection rate by 4% on the CIFAR10 dataset and by more than 50% for the EMNIST dataset.

Abstract Context In the last decade of data-driven decision-making, Machine Learning (ML) systems reign supreme. Because of the different characteristics between ML and traditional Software Engineering systems, we do not know to what extent the issue-reporting needs are different, and to what extent these differences impact the issue resolution process. Objective We aim to compare the differences between ML and non-ML issues in open-source applied AI projects in terms of resolution time and size of fix. This research aims to enhance the predictability of maintenance tasks by providing valuable insights for issue reporting and task scheduling activities. Method We collect issue reports from Github repositories of open-source ML projects using an automatic approach, filter them using ML keywords and libraries, manually categorize them using an adapted deep learning bug taxonomy, and compare resolution time and fix size for ML and non-ML issues in a controlled sample. Result 147 ML issues and 147 non-ML issues are collected for analysis. We found that ML issues take more time to resolve than non-ML issues, the median difference is 14 days. There is no significant difference in terms of size of fix between ML and non-ML issues. No significant differences are found between different ML issue categories in terms of resolution time and size of fix. Conclusion Our study provided evidence that the life cycle for ML issues is stretched, and thus further work is required to identify the reason. The results also highlighted the need for future work to design custom tooling to support faster resolution of ML issues.

Containers are an increasingly used mechanism for providing low-cost, lightweight, portable, standalone application deployments, particularly for service orchestration. Docker provides container technology that enables a single host to isolate several applications and deploy them rapidly in different environments. The increasing demand for container applications and the growing popularity of Docker has motivated extensive research into evaluating the performance, energy consumption, and running cost of Docker-based computation. This paper investigates the energy footprint of Docker containers and workloads. To motivate research in energy-efficient container development, this paper takes a practical approach to measure the energy consumption in common Docker containers under various workloads.

It is generally accepted in Software Engineering that code clones – often the result of copy-and-paste of existing code – result in poorer maintainability of software systems. Consequently, a variety of techniques have been devised to detect cloned code in software systems and alert developers of duplicated code. Most techniques operate at the source-code level and require some combination of pretty-printing, tokenization and abstraction in order to improve the comparison of code fragments over purely string-based techniques. Avoiding some of the issues of source-code based approaches, we are investigating the effectiveness of using various similarity measures on Bytecode to identify code clones in Java-based systems in this work. The results of our evaluation on selected Java systems indicate that instruction sequences can be used to effectively detect identical code clones. Especially, we achieved the best performance when using the normalized edit distance among applied similarity measures.

Having precise specifications of service APIs is essential for many Software Engineering activities. Unfortunately, available documentation of services is often inadequate and/or imprecise and, hence, cannot be fully relied upon. Generating service documentation manually is a tedious and error-prone task, especially in light of changes to services. Therefore, there is a need for automated support in generating service documentation. In this work, we present a novel approach to infer the API of a service by analyzing recorded messages sent to and received from this service. Our approach includes a novel, two-level clustering technique to cluster messages, a step that many existing approaches to infer message formats fail to perform precisely in the presence of significant variation of payload information of the available messages. We have evaluated our approach on message traces from four different real-world services. The experimental result shows that our approach is more effective than existing techniques in extracting correct message formats from recorded messages.

In supporting online student cohorts, we experienced challenges in achieving the same quality of engagement asynchronously as we do through face-to-face discussions in a classroom setting. The educational model we use is based upon students progressing through weekly tasks designed to support development, and measure achievement of learning outcomes. In this model, once a student has completed a task, interactions between the student and instructor provide feedback to the instructor of the student's understanding. The feedback system gives confidence that the student understands their work and aids in identifying learning intervention opportunities. To achieve this asynchronously, we developed and integrated an audio-discussion tool known as Real Talk into our Learning Management System (LMS). The tool allows instructors to record discussion prompts tailored to a student's completed task and has the LMS replay the prompt(s) and immediately capturing the student's response. These interactions replicate essential aspects of face-to-face, in-person discussions by not affording the student opportunities to research and rehearse responses, which we previously experienced when using asynchronous discussion or quiz tools for this purpose. In this paper, we present the implementation of the Real Talk tool and discuss results evaluating how effective it was at allowing instructors to identify opportunities for learning interventions in introductory computing courses. The results confirmed that the tool has assisted in identifying knowledge gaps not identifiable in students' submissions alone.

APIs play a significant role in the sharing, utilization and integration of information and service assets for enterprises, delivering significant business value. However, the documentation of service APIs can often be incomplete, ambiguous, or even non-existent, hindering API-based application development efforts. In this paper, we introduce an approach to automatically mine the fine-grained message formats required in defining the APIs of services and applications from their interaction traces, without assuming any prior knowledge. Our approach includes three major steps with corresponding techniques: (1) classifying the interaction messages of a service into clusters corresponding to message types, (2) identifying the keywords of messages in each cluster, and (3) extracting the format of each message type. We have applied our approach to network traces collected from four real services which used the following application protocols: REST, SOAP, LDAP and SIP. The results show that our approach achieves much greater accuracy in extracting message formats for service APIs than current state-of-art approaches.

Writing a good usability defect report can be a tedious task, especially in identifying what important information should be included, and capturing the attention of software developers to fix them. This paper is a continuity of our previous studies investigating software development practitioners' day-to-day practices when dealing with usability defects. In this study, we mined 377 developer-tagged usability defect reports from Mozilla Thunderbird, Firefox for Android and Eclipse Platform to confirm what software development practitioners claimed to provide when reporting usability defects. We looked for the presence of key defect attributes - steps to reproduce, impact, software context, expected output, actual output, assumed causes, solution proposal and supplementary information. In addition, we analyzed the trend of different types of usability defects, correlation between usability defects and defect severity, and failure qualifier. Our findings demonstrate a mismatch between what software development practitioners claimed to provide when reporting usability defects, and the information that actually appears in the defect reports. The results of our research have important implications for software defect reporting, especially in designing more effective mechanisms for reporting usability defects.

The landscape of modern enterprise IT environments is that a large number of distributed software systems interact and cooperate with each other to support daily business operations. With the prevalence of cloud computing, the level of system connectivity increases further. The large scale of such an environment makes it difficult to test a system's quality attributes such as performance and scalability before it is actually deployed in the production environment. Under the currently dominant iterative and incremental software development paradigm, this difficulty is even more pronounced when the quality attributes of an enterprise system need to be examined and evaluated at early stages but a large part of its operating environment is not available or accessible. In this paper, we present a Coloured Petri nets (CPN) based system behaviour emulation approach and a lightweight emulated testing framework for provisioning a virtual deployment testing environment for an enterprise software system, so that its quality attributes, especially scalability, can be evaluated without physically connecting to the real production environment. It is worth noting that the focus of this work is on the testing environment which enables virtual system deployment and testing, instead of being on the research topic of deployment test. CPN and its associated design and simulation tools have been used and integrated to model and execute the behaviour of those cooperating endpoint systems that an enterprise software system interacts with. We have successfully implemented an industry standard protocol Lightweight Directory Access Protocol in our approach and applied it in testing the scalability of a real-world enterprise application, CA Technologies’ IdentityManager. A thorough in-lab performance study has also been conducted to examine the capacity and scalability of this approach.

Software development organizations invest in test automation tools and methods to optimize defect discovery rates. The true value of these tools is realized when the defects are addressed before release, and hence good quality defect reports are critical. We describe a survey we conducted to better understand usability defect reporting, in particular, influences on the quality of usability defect reports. We analyze feedback from nearly 150 software developers and usability defect reporters and identify key determinants of quality defect reports, aspects of usability defects that are challenging to report and directions for future research into usability defect reporting tools to improve usability defect reports quality.

Finding the best deployment configuration that maximises energy efficiency while guaranteeing system performance of cloud applications is an extremely challenging task. It requires the evaluation of system performance and energy consumption under a wide variety of realistic workloads and deployment configurations. This paper demonstrates StressCloud, an automatic performance and energy consumption analysis tool for cloud applications in real-world cloud environments. StressCloud supports 1) the modelling of realistic cloud application workloads, 2) the automatic generation and running of load tests, and 3) the profiling of system performance and energy consumption.

Failure of application operations is one of the main causes of system-wide outages in cloud environments. This particularly applies to DevOps operations, such as backup, redeployment, upgrade, customized scaling, and migration that are exposed to frequent interference from other concurrent operations, configuration changes, and resources failure. However, current practices fail to provide a reliable assurance of correct execution of these kinds of operations. In this paper, we present an approach to address this problem that adopts a regression-based analysis technique to find the correlation between an operation's activity logs and the operation activity's effect on cloud resources. The correlation model is then used to derive assertion specifications, which can be used for runtime verification of running operations and their impact on resources. We evaluated our proposed approach on Amazon EC2 with 22 rounds of rolling upgrade operations while other types of operations were running and random faults were injected. Our experiment shows that our approach successfully managed to raise alarms for 115 random injected faults, with a precision of 92.3%.

In cloud environments, IT solutions are delivered to users via shared infrastructure, enabling cloud service providers to deploy applications as services according to user QoS (Quality of Service) requirements. One consequence of this cloud model is the huge amount of energy consumption and significant carbon footprints caused by large cloud infrastructures. A key and common objective of cloud service providers is thus to develop cloud application deployment and management solutions with minimum energy consumption while guaranteeing performance and other QoS specified in Service Level Agreements (SLAs). However, finding the best deployment configuration that maximises energy efficiency while guaranteeing system performance is an extremely challenging task, which requires the evaluation of system performance and energy consumption under various workloads and deployment configurations. In order to simplify this process we have developed Stress Cloud, an automatic performance and energy consumption analysis tool for cloud applications in real-world cloud environments. Stress Cloud supports the modelling of realistic cloud application workloads, the automatic generation of load tests, and the profiling of system performance and energy consumption. We demonstrate the utility of Stress Cloud by analysing the performance and energy consumption of a cloud application under a broad range of different deployment configurations.

In cloud environments, IT solutions are delivered to users via shared infrastructure. One consequence of this model is that large cloud data centres consume large amounts of energy and produce significant carbon footprints. A key objective of cloud providers is thus to develop resource provisioning and management solutions at minimum energy consumption while still guaranteeing Service Level Agreements (SLAs). However, a thorough understanding of both system performance and energy consumption patterns in complex cloud systems is imperative to achieve a balance of energy efficiency and acceptable performance. In this paper, we present StressCloud, a performance and energy consumption analysis tool for cloud systems. StressCloud can automatically generate load tests and profile system performance and energy consumption data. Using StressCloud, we have conducted extensive experiments to profile and analyse system performance and energy consumption with different types and mixes of runtime tasks. We collected fine-grained energy consumption and performance data with different resource allocation strategies, system configurations and workloads. The experimental results show the correlation coefficients of energy consumption, system resource allocation strategies and workload, as well as the performance of the cloud applications. Our results can be used to guide the design and deployment of cloud applications to balance energy and performance requirements.

When operating in volatile environments, service-based systems (SBSs) that are dynamically composed from component services must be monitored in order to guarantee timely and successful delivery of outcomes in response to user requests. However, monitoring consumes resources and very often impacts on the quality of the SBSs being monitored. Such resource and system costs need to be considered in formulating monitoring strategies for SBSs. The critical path of a composite SBS, i.e., the execution path in the service composition with the maximum execution time, is of particular importance in cost-effective monitoring as it determines the response time of the entire SBS. In volatile operating environments, the critical path of an SBS is probabilistic, as every execution path can be critical with a certain probability, i.e., its criticality. As such, it is important to estimate the criticalities of different execution paths when deciding which parts of the SBS to monitor. Furthermore, cost-effective monitoring also requires management of the trade-off between the benefit and cost of monitoring. In this paper, we propose CriMon, a novel approach to formulating and evaluating monitoring strategies for SBSs. CriMon first calculates the criticalities of the execution paths and the component services of an SBS and then, based on those criticalities, generates the optimal monitoring strategy considering both the benefit and cost of monitoring. CriMon has two monitoring strategy formulation methods, namely local optimisation and global optimisation. In-lab experimental results demonstrate that the response time of an SBS can be managed cost-effectively through CriMon-based monitoring. The effectiveness and efficiency of the two monitoring strategy formulation methods are also evaluated and compared.

Cloud computing delivers IT solutions as a utility to users. One consequence of this model is that large cloud data centres consume large amounts of energy and produce significant carbon footprints. A common objective of cloud providers is to develop resource provisioning and management solutions that minimise energy consumption while guaranteeing Service Level Agreements (SLAs). In order to achieve this objective, a thorough understanding of energy consumption patterns in complex cloud systems is imperative. We have developed an energy consumption model for cloud computing systems. To operationalise this model, we have conducted extensive experiments to profile the energy consumption in cloud computing systems based on three types of tasks: computation-intensive, data-intensive and communication-intensive tasks. We collected fine-grained energy consumption and performance data with varying system configurations and workloads. Our experimental results show the correlation coefficients of energy consumption, system configuration and workload, as well as system performance in cloud systems. These results can be used for designing energy consumption monitors, and static or dynamic system-level energy consumption optimisation strategies for green cloud computing systems.

Enterprise software systems operate in large-scale, heterogeneous, distributed environments which makes assessment of non-functional properties, such as scalability and robustness, of those systems particularly challenging. Enterprise environment emulators can provide test-beds representative of real environments using only a few physical hosts thereby allowing assessment of the non-functional properties of enterprise software systems. To date, analysing outcomes of these tests has been an ad hoc and somewhat tedious affair; largely based on manual and/or script-assisted inspection of interaction logs. Quokka visualises emulations significantly aiding analysis and comprehension. Emulated interactions can be viewed live (in real-time) as well as be replayed at a later stage, furthermore, basic charts are used to aggregate and summarise emulations, helping to identify performance and scalability issues.

The loss of control over information assets is a major security and privacy concern in the Cloud. Service consumers typically have no insights which controls protect their information assets and how effectively. To tackle this challenge, we propose an approach where service providers and consumers conjointly manage security requirements for a Cloud service following the ISO 27001 standard for information security management. We have developed a security management platform that provides tool support for service providers and consumers (i) to specify and consolidate security requirements and (ii) to collect, measure, analyse and report information about the effectiveness of implemented controls. By involving service consumers in management activities following an international standard, our approach helps service providers to increase transparency and traceability of their security measures whereas service consumers gain much-needed insights in the protection of their information assets. The applicability of our approach is demonstrated with an example scenario.

Information technology is playing a more and more critical role in many large organizations and enterprise software systems have become increasingly integrated to better support the goals of these organizations. As a consequence, it is a major engineering challenge to assure quality of a software system that is to be deployed in an enterprise environment containing many thousands of interconnected systems. To address this challenge, we propose the use of a scalable emulation environment enabling real-time system testing in large-scale settings. In this work, we discuss the main concepts of our approach, present Reac2o, a prototype implementation of such an emulation environment, and illustrate its applicability in the context of identity management.

Real software systems change and become more complex over time. But which parts change and which parts remain stable? Common wisdom, for example, states that in a well-designed object-oriented system, the more popular a class is, the less likely it is to change from one version to the next, since changes to this class are likely to impact its clients. We have studied consecutive releases of several public domain, object-oriented software systems and analyzed a number of measures indicative of size, popularity, and complexity of classes and interfaces. As it turns out, the distributions of these measures are remarkably stable as an application evolves. The distribution of class size and complexity retains its shape over time. Relatively little code is modified over time. Classes that tend to be modified, however, are also the more popular ones, that is, those with greater Fan-In. In general, the more "complex" a class or interface becomes, the more likely it is to change from one version to the next.

The requirement for 24/7 availability of distributed applications complicates their maintenance and evolution as shutting down such applications to perform updates may not be an acceptable solution. Therefore, there is a need to be able to update these applications dynamically, i.e. without shutting them down. Current solutions for building dynamically updatable Java applications require that applications either are prepared for updates from the outset, comply with a specific framework, or are executed in a modified virtual machine. In this work, we present a novel approach to creating dynamically updatable Java applications based on the concept of partitioning applications into units of dynamic updates and illustrate how this approach better addresses the problems of adding update support to existing applications than traditional approaches.

To address the problems of traditional software development, recent years have shown the introduction of more light-weight or "agile" development processes (eXtreme Programming being the most prominent one). These processes are intended to support early and quick production of working code by structuring the development into small release cycles and focus on continual interaction between developers and customers. As such software development processes become more popular there is a growing demand from industry to introduce agile development practices in tertiary education. This is not a straightforward task as the corresponding practices may run counter to educational goals or may not be adjusted easily to a learning environment. In this paper, we discuss some of these issues and reflect on the problems of teaching agile processes in tertiary education.