The overtake car maneuver involves high risk and complex judgement. For autonomous vehicles this is challenging, especially for human-initiated overtake requests. If a user requests the maneuver there must be a rapid safety assessment. Language models have great potential to classify safety with explanations, but they struggle to disentangle critical information from complex vehicular environments. We apply CLEAR (Cue Learning using Evolution for Accurate Recognition) to evolve prompt cues that optimize the ability of language models to correctly predict safety scores for overtaking maneuvers. To achieve this, we create a novel open-source symbolic traffic model EvoDrive designed specifically for EC research, which outputs LLM-readable snapshots. We show that LLMs + EvoDrive with CLEAR can reduce error by more than 20% compared to without CLEAR, with statistically significant results. Analysis shows evolved cues are coherent and have reduced variability in LLM output.

This paper presents a multimodal human-robot interaction (HRI) system for educational contexts implemented on the humanoid robot Pepper. The system leverages multiple communicative channels, allowing learners to combine speech with tablet interaction while the robot responds through synchronized speech, textual captions and dynamic visual cues. To ensure robustness and reliability, we introduce the use of Metamorphic Testing for multimodal HRI. By validating system behavior through systematic input transformations, we demonstrate how metamorphic testing can uncover inconsistencies across linguistic, visual and cross-modal interactions. This work contributes both a novel methodological framework for evaluating multimodal HRI systems and an application to educational robotics.

Large Language Model (LLM) image recognition is a powerful tool for extracting data from images, but accuracy depends on providing sufficient cues in the prompt - requiring a domain expert for specialized tasks. We introduce Cue Learning using Evolution for Accurate Recognition (CLEAR), which uses a combination of LLMs and evolutionary computation to generate and optimize cues such that recognition of specialized features in images is improved. It achieves this by auto-generating a novel domain-specific representation and then using it to optimize suitable textual cues with a genetic algorithm. We apply CLEAR to the real-world task of identifying sustainability data from interior and exterior images of buildings. We investigate the effects of using a variable-length representation compared to fixed-length and show how LLM consistency can be improved by refactoring from categorical to real-valued estimates. We show that CLEAR enables higher accuracy compared to expert human recognition and human-authored prompts in every task with error rates improved by up to two orders of magnitude and an ablation study evincing solution concision.

Deep Neural Networks (DNNs) have become a core component in several safety-critical tasks like Autonomous Driving. To ensure their safe usage in these scenarios, DNN repair approaches have been used to fix erroneous predictions by changing the values of a small subset of parameters like model weights. However, in order to work effectively, these approaches require the presence of categorical outputs, for which it is possible to provide a clear assessment of correctness. Therefore, they do not work well for regression networks for which such binary assessment is not possible. To solve this issue, we propose a new approach, called Semsegrep, to enable practitioners to repair regression networks like semantic segmentation, which are increasingly deployed in these safety-critical domains. Semseg-repfirst selects weights that contribute to erroneous predictions, consequently decreasing a target metric like Mean Intersection over Union. It then employs a filtering method that selects a subset of these weights based on their determined potential for repairing the model. Lastly, it uses a search-based repair approach to generate a patch with new value assignments for the selected weights. Experiments conducted on a semantic segmentation model and four datasets provided by our industry partner show that Semsegrep is able to improve the model according to the given target metric without affecting the overall accuracy, and it is better than a state-of-the-art repair approach.

Sufficient testing against real-world ML model is mandatory because the safety violation of that behavior causes a huge social impact. Existing methods detect weaknesses in ML models using labeled features associated with data based on the concept of combinatorial test modeling. However, sufficient testing is difficult due to rare features that are difficult to collect from the real world, especially for image data. In this paper, we investigate an approach using an image generation model to solve this problem, and answer the following research question: To what extent can the generated data detect weaknesses by factor-level compared to the test data? Our experiments showed that $74 \%$ of the weaknesses detected by the generated data were detected by the test data as well. Our further investigations showed that the feasibility of generating data in a way that transforms factor levels is influenced by the generative model used for the transformation and the type of transformation.

Cyber-physical systems (CPSs) are increasingly adopting deep neural networks (DNNs) as controllers, giving birth to AI-enabled CPSs . Despite their advantages, many concerns arise about the safety of DNN controllers. Numerous efforts have been made to detect system executions that violate safety specifications; however, once a violation is detected, to fix the issue, it is necessary to localise the parameters of the DNN controller responsible for the wrong decisions leading to the violation. This is particularly challenging, as it requires to consider a sequence of control decisions, rather than a single one, preceding the violation. To tackle this problem, we propose SpectAcle , that can localise the faulty parameters in DNN controllers. SpectAcle considers the DNN inferences preceding the specification violation and uses forward impact to determine the DNN parameters that are more relevant to the DNN outputs. Then, it identifies which of these parameters are responsible for the specification violation, by adapting classic suspiciousness metrics. Moreover, we propose two versions of SpectAcle , that consider differently the timestamps that precede the specification violation. We experimentally evaluate the effectiveness of SpectAcle on 6,067 faulty benchmarks, spanning over different application domains. The results show that SpectAcle can detect most of the faults.

Conceptual architecture involves a highly creative exploration of novel ideas, often taken from other disciplines as architects consider radical new forms, materials, textures and colors for buildings. While today's generative AI systems can produce remarkable results, they lack the creativity demonstrated for decades by evolutionary algorithms. SCAPE, our proposed tool, combines evolutionary search with generative AI, enabling users to explore creative and good quality designs inspired by their initial input through a simple point and click interface. SCAPE injects randomness into generative AI, and enables memory, making use of the built-in language skills of GPT -4 to vary prompts via text-based mutation and crossover. We demonstrate that compared to DALL. E 3, SCAPE enables a 67% improvement in image novelty, plus improvements in quality and effectiveness of use; we show that in just three iterations SCAPE has a 24% image novelty increase enabling effective exploration, plus optimization of images by users. We use more than 20 independent architects to assess SCAPE, who provide markedly positive feedback.

In AI-enabled CPSs, DNNs are used as controllers for the physical system. Despite their advantages, DNN controllers can produce wrong control decisions, which can lead to safety risks for the system. Once wrong behaviors are detected, the DNN controller should be fixed. DNN repair is a technique that allows to perform this fine-grained improvement. However, state-of-the-art DNN repair techniques require ground-truth labels to guide the repair. For AI-enabled CPSs, these are not available, as it is not possible to assess whether a specific control decision is correct. Nevertheless, it is possible to assess whether the DNN controller leads to wrong behaviors of the controlled system by considering system-level requirements. In this paper, following this observation, we propose a novel DNN repair approach that is guided by system-level specifications. The approach takes in input a system-level specification, some tests violating the specification, and some faulty DNN weights. The approach searches for alternative weight values with the goal of fixing the behavior on the failing tests without breaking the passing tests. We also propose a heuristic that allows us to accelerate the search by avoiding the execution of some tests. Experiments on real-world AI-enabled CPSs show that the approach effectively repairs their controllers.

Delivery systems operated by autonomous robots use schedulers to allocate robots to the different orders. Such schedulers are often optimisation-based algorithms that aim to maximise the number of delivered goods. The oracle problem affects the testing of these schedulers, as it is not always possible to assess whether the schedule produced for a given scenario is the optimal one. In this work, we propose a framework, based on a novel use of metamorphic testing, to assess the optimality of the scheduling algorithm developed by Panasonic for the management of a fleet of autonomous delivery robots in the Fujisawa Sustainable Smart Town, Japan. In the framework, a metamorphic relation (MR) transforms a source test case in a followup test case in a predefined way, and compares the results of the execution of the two tests in a simulated environment: if the comparison violates the expected relation, we can claim that one of the two schedules produced by the scheduler is suboptimal. We propose 19 MRs that target different aspects of the delivery system. Experiments over more than 900,000 test cases show that the different MRs have different abilities in exposing suboptimal behaviour and that most of the MRs do not subsume each other. Moreover, they also show that MR violations can provide useful insights into the scheduler's behaviour to Panasonic's engineers.

We introduce a goal-aware extension of responsibility-sensitive safety (RSS), a recent methodology for rule-based safety guarantee for automated driving systems (ADS). Making RSS rules guarantee goal achievement—in addition to collision avoidance as in the original RSS—requires complex planning over long sequences of manoeuvres. To deal with the complexity, we introduce a compositional reasoning framework based on program logic, in which one can systematically develop RSS rules for smaller subscenarios and combine them to obtain RSS rules for bigger scenarios. As the basis of the framework, we introduce a program logic dFHL that accommodates continuous dynamics and safety conditions. Our framework presents a dFHL-based workflow for deriving goal-aware RSS rules; we discuss its software support, too. We conducted experimental evaluation using RSS rules in a safety architecture. Its results show that goal-aware RSS is indeed effective in realising both collision avoidance and goal achievement.

Autonomous robots are emerging as a solution to various challenges of last mile goods delivery, like reducing traffic congestion, pollution, and costs. The configuration of an autonomous delivery robots system requires balancing aspects like delivery rate, cost of robots' operation, and required monitoring efforts. Our industry partner Panasonic is employing a search-based approach to find the configurations of the system that optimise these three aspects for a given set of customers' orders. The approach uses a simulator to assess the different configurations in the fitness functions' computation. Due to the high cost of the simulation, the whole search-based approach is computationally expensive. A classic approach to speed up such approaches is to use surrogate models trained on example simulation data that allow to approximate the results of a simulated configuration with negligible computational cost. A risk when using such approaches is to underestimate the cost of building the surrogate model itself, that can exceed the computational gain obtained during the search, thus making the adoption of surrogate models detrimental. In this work, we propose an approach in which the surrogate model is not trained before the search; instead, the approach alternates between training the model on subsets of data of increasing size, and searching using these cheaper models until the search stagnates. Experiments over 144,000 settings of the search show that the proposed approach can significantly reduce the cost of searching for configurations, while having an acceptable impact on the Quality of the configurations it finds.

Finding optimal configurations for complex systems, such as a fleets of autonomous delivery robots, is a complex task that benefits from automation. Automated search-based approaches have been proposed to automatically find such configurations. Although the configurations found by these methods perform well on average, they may be non-stable, i.e., their performance could vary greatly across scenarios. When deploying a system with a given configuration, it is important to know that it will perform adequately for the range of possible scenarios, i.e., to reduce how much the system's performance varies between scenarios. To this end, we attempt to make the search-based approaches aware of the configurations' stability. We explore two ways of doing this: by integrating it into the fitness functions describing the target performance metrics, and by adding it as a separate set of additional objectives. We applied the two approaches to find optimal configurations of a fleet of robots for automatic delivery service. Results show that integrating the stability concern into the fitness functions is better than treating it separately.

Autonomous robots can solve different issues of delivery services, by guaranteeing less traffic congestion, less pollution, and lower operational costs. Designing such type of delivery system based on autonomous robots requires the collaboration of different stakeholders, having different concerns: the store utilising the delivery service that is interested in costs and customer satisfaction, the municipality where the service is operated that is interested in the safety of the service, and the robotic company providing the service that is interested in all previous concerns. Our industrial partner from the robotic domain is designing this type of service in a smart town, and using a simulator for assessing different configurations providing different levels of performance. Since manually designing the configurations is time consuming for engineers, in this paper, we propose a search-based approach (All) that is able to explore the space of service configurations and find the optimal ones that show the tradeoff existing among the different concerns, so that stakeholders can make an informed decision. Since assessing one configuration requires to simulate the service multiple times over different types of customer requests, the approach suffers from scalability issues. Therefore, we propose two improvements of the approach that reduce the number of required simulations (IncrSim), and the duration of the simulation (IncrTime). Ex-periments on different settings show that IncrSim and IncrTime can find results as good as those of All in less time, and better than versions of All executed for the same budget.

The use of autonomous robots for delivery of goods to customers is an exciting new way to provide a reliable and sustainable service. However, in the real world, autonomous robots still require human supervision for safety reasons. We tackle the real-world problem of optimizing autonomous robot timings to maximize deliveries, while ensuring that there are never too many robots running simultaneously so that they can be monitored safely. We assess the use of a recent hybrid machine-learning-optimization approach COIL (constrained optimization in learned latent space) and compare it with a baseline genetic algorithm for the purposes of exploring variations of this problem. We also investigate new methods for improving the speed and efficiency of COIL. We show that only COIL can find valid solutions where appropriate numbers of robots run simultaneously for all problem variations tested. We also show that when COIL has learned its latent representation, it can optimize 10% faster than the GA, making it a good choice for daily re-optimization of robots where delivery requests for each day are allocated to robots while maintaining safe numbers of robots running at once.

Deep Neural Networks (DNNs) are finding a place at the heart of more and more critical systems, and it is necessary to ensure they perform in as correct a way as possible. Search-based repair methods, that search for new values for target neuron weights in the network to better process fault-inducing inputs, have shown promising results. These methods rely on fault localisation to determine what weights the search should target. However, as the search progresses and the network evolves, the weights responsible for the faults in the system will change, and the search will lose in effectiveness. In this work, we propose an adaptive search method for DNN repair that adaptively updates the target weights during the search by performing fault localisation on the current state of the model. We propose and implement two methods to decide when to update the target weights, based on the progress of the search's fitness value or on the evolution of fault localisation results. We apply our technique to two image classification DNN architectures against a dataset of autonomous driving images, and compare it with a state-of-the art search-based DNN repair approach.

Autonomous robots have recently been employed for goods delivery, with the goal of reducing traffic congestion, pollution, and operational costs. The design of such a delivery service requires to select the number of robots, their operating hours, and speed. Requirements from different stakeholders must be considered: customer satisfaction, cost, and safety. To assist with said design, our industry partner Panasonic is employing a search-based approach that tries to find service configurations that optimise the three requirements, on average, across different possible sets of customer requests. The obtained Pareto fronts of solutions show the trade-off existing among the different requirements. Such Pareto fronts, albeit very useful, do not always facilitate an informed decision for the stakeholders, for they provide too many solutions (some of them very similar to each other). To tackle this issue, in this paper we propose two approaches to prune and simplify Pareto fronts. Our approaches consider the standard deviation of objective values across the different sets of customer requests; the intuition is that, if two solutions (expressed in terms of average objective values) overlap based on their standard deviations, they can be considered similar. Based on this intuition, the two pruning approaches group similar solutions and select only one representative for each partition. We assessed these pruning methods on the Pareto fronts obtained with the search-based approach employed by Panasonic. We found that they can significantly reduce the size of the Pareto fronts while retaining a reasonable amount of their unpruned quality (measured in terms of Hypervolume).

Deep Neural Networks (DNNs) are applied in several safety-critical domains and their trustworthiness is of paramount importance. For example, DNNs used in autonomous driving as classifiers should not misclassify detected objects; however, since obtaining perfect accuracy is not possible, special attention should be given to the most critical cases, e.g., pedestrians. This has been confirmed by the consortium of our partners from the automotive domain that provided us with specific risk levels for different misclassifications. A recent approach to improve DNN performance is to localise DNN weights responsible for the misclassifications and then adjust (repair) them to improve the misclassifications. However, they under-perform when they need to consider multiple misclassifications, and they do not consider the risk levels of the different misclassifications. To tackle this, we propose DISTRREP, a distributed repair approach that first finds the best fixes for each critical misclassification, and then integrates them in a single repaired DNN model, by considering the risk levels. We assess DISTRREP over three DNN models and a dataset of autonomous driving images, by considering requirements specified by our industrial partners. Experiments show that DISTRREP is more effective than baseline approaches based on retraining, and other risk-unaware repair approaches.

The recent fast development of quantum computers breaks several computation limitations that are difficult for conventional computers. Up to the present, although many approaches and tools have been proposed to test quantum programs, the fundamental features of quantum programs, i.e., magnitude, phase, and entanglement, have been largely overlooked, leading to limited fault detection capability and reduced testing effectiveness. To address this problem, we propose an automated testing framework named QURATEST, equipped with three test case generators (including two newly proposed techniques, UCNOT and IQFT in this paper, as well as one based on Random techniques) to test quantum programs. Overall, the proposed generators enable the generation of diverse test inputs by considering the quantum features of quantum programs. In the experiments, we perform an in-depth evaluation of QURATEST from three aspects: generated test case diversity, output coverage of the program under test, and fault detection capability. The results demonstrate the potential of our newly proposed techniques in that IQFT can generate the most diverse test cases regarding magnitude, phase, and entanglement, with 66% cell coverage. Comparatively, the Random approach only has 10% cell coverage. Regarding the evaluations of the output coverage, IQFT can achieve the highest output coverage in 70.2% (33 out of 47) of all quantum programs. In terms of fault detection, UCNOT outperforms the other two techniques. Specifically, the test cases generated by UCNOT have the best mutation score in 88.4% (23 out of 26) quantum programs.

Domain-specific modeling (DSM) is used as an approach to further improve the efficiency of MBD (Model-based development). While DSM improves model description efficiency, it is necessary to construct a modeling environment that supports DSM for each domain. The DSM environment construction cost is an obstacle to its widespread use. In this paper, we propose RapidDSM, a DSM environment construction support tool that enumerates model representations based on patterns, to support the DSM environment construction. By enumerating feasible views from meta-models based on patterns, the tool enables rapid exploration of various model representation designs. To validate the effectiveness of this tool, we evaluated our method by applying EAST-ADL, an example of a published DSM. As a result of the evaluation, it was confirmed whether the DSM view was enumerated from the defined EAST-ADL DSM metamodel and whether a practical DSM editor could be built.

Systems based on Deep Neural Networks (DNNs) are increasingly being used in industry. In the process of system operation, DNNs need to be updated in order to improve their performance. When updating DNNs, systems used in companies that require high reliability must have as few regressions as possible. Since the update of DNNs has a data-driven nature, it is difficult to suppress regressions as expected by developers. This paper identifies the requirements for DNN updating in industry and presents a case study using techniques to meet those requirements. In the case study, we worked on satisfying the requirement to update models trained on car images collected in Fujitsu assuming security applications without regression for a specific class. We were able to suppress regression by customizing the objective function based on NeuRecover, a DNN repair technique. Moreover, we discuss some of the challenges identified in the case study.

Autonomous Driving Systems (ADSs) are complex systems that must consider different aspects such as safety, compliance to traffic regulations, comfort, etc. The relative importance of these aspects is usually balanced in a weighted cost function. However, there is generally no optimal set of weights, and different driving situations may require different weights values to guarantee a safe drive. Recent testing approaches can generate diverse driving scenarios in which different ADS configurations lead to various degrees of hazard. These tests need to be properly analyzed to improve the ADS's safety. In this paper, we propose an analysis approach that is able to assess the relation between the ADS configurations and the level of hazard that is obtained in some particular traffic situations. The approach uses fuzzification to partition ADS weights in different categories, and a spectrum-based analysis to identify which weights categories are related to hazard and safety. The occurrence of a hazard could be due to a single weight or to combinations of two or more weights. For scalability, the approach performs an incremental analysis, in which first single weights are considered, and then weight combinations of higher order. The approach has been applied to an industrial path planner.

Autonomous Driving Systems (ADSs) are promising, but must show they are secure and trustworthy before adoption. Simulation-based testing is a widely adopted approach, where the ADS is run in a simulated environment over specific scenarios. Coverage criteria specify what needs to be covered to consider the ADS sufficiently tested. However, existing criteria do not guarantee to exercise the different decisions that the ADS can make, which is essential to assess its correctness. ADSs usually compute their decisions using parameterised rule-based systems and cost functions, such as cost components or decision thresholds. In this article, we argue that the parameters characterise the decision process, as their values affect the ADS’s final decisions. Therefore, we propose parameter coverage, a criterion requiring to cover the ADS’s parameters. A scenario covers a parameter if changing its value leads to different simulation results, meaning it is relevant for the driving decisions made in the scenario. Since ADS simulators are slightly uncertain, we employ statistical methods to assess multiple simulation runs for execution difference and coverage. Experiments using the Autonomoose ADS show that the criterion discriminates between different scenarios and that the cost of computing coverage can be managed with suitable heuristics.

Additional training of a deep learning model can cause negative effects on the results, turning an initially positive sample into a negative one (degradation). Such degradation is possible in real-world use cases due to the diversity of sample characteristics. That is, a set of samples is a mixture of critical ones which should not be missed and less important ones. Therefore, we cannot understand the performance by accuracy alone. While existing research aims to prevent a model degradation, insights into the related methods are needed to grasp their benefits and limitations. In this talk, we will present implications derived from a comparison of methods for reducing degradation. Especially, we formulated use cases for industrial settings in terms of arrangements of a data set. The results imply that a practitioner should care about better method continuously considering dataset availability and life cycle of an AI system because of a trade-off between accuracy and preventing degradation.

Simulation-based testing is a popular approach for testing autonomous driving systems (ADS), in which different types of scenario are designed to test the ADS under different driving conditions. Given a specific test goal, a generation approach (e.g., search-based testing) is usually employed to find a scenario covering such goal; for example, it can find a scenario in which the autonomous vehicle collides. The generated scenarios may contain some elements that are irrelevant for the achievement of the test goal; if this is the case for a scenario that exposes a failure, for ADS engineers it is difficult to identify the root cause, as the ADS interacts with several traffic participants and it is not clear which of these are essential to trigger the failure. This problem emerged during the collaboration with our industry partner, for which, in the past, we proposed different test generation approaches for their ADS path planner, but these may produce test scenarios that are not minimal. To tackle this problem, in this paper, we propose an approach that, given an ADS test scenario, simplifies it by removing all the traffic participants that are not needed. As output, the approach provides a scenario that still covers the test goal as the initial scenario, but contains the minimum number of traffic participants. The approach consists in iteratively generating simplified scenarios by removing some traffic participants and determining, by observing the test execution, which can be actually removed and which must be kept in the scenario. Three policies are investigated to remove traffic participants whose classification is not know: single policy, binary policy, and adaptive policy. Experiments have been conducted on several scenarios generated for the path planner. Results show that the binary policy is the one that usually can find the minimal scenario with the minimum number of simplification attempts, but, in particular cases, the adaptive policy is better.

The difficulty in exploring the game balance has been increasing, especially in Game-as-a-Service (GaaS) with updates in every few weeks, and due to the complexity in game design and business models. In the limited time available for testing, using automated game agents enables much more test plays than using human test players does, and it has been accelerated by the recent progress of deep reinforcement learning. However, understanding specific behaviours of each agent is hard due to their “black-box” nature. In this paper, we propose a method for explaining the behaviour of game agents using differential comparison between agents. This comparison approach is motivated by our experience with existing explanation techniques that often extracted uninteresting, common aspects of the behaviour. In addition, there are large potentials for the application of the comparison: between agents with different learning algorithms, between human agents and automated agents, and between test agents and users. We applied our technique to a prototype of a commercial GaaS and confirmed our technique can extract specific differences between agents.

Autonomous Driving Systems (ADSs) are complex systems that must satisfy multiple safety requirements. In particular cases, all the requirements cannot be satisfied at the same time, and the control software of the ADS must make trade-offs among their satisfaction. Usually, the trading-offs in the decision-making process are configurable; different configuration options can affect driving behaviors, satisfying or violating requirements at different degrees. Therefore, it is highly important to know whether a configuration can guarantee a safe drive or not, i.e., whether it leads to requirement violations that exceed the allowable range or not. However, there is currently no approach to systematically assess the safety of ADS configurations from the perspective of requirements violations. To bridge this gap, this paper proposes a “Hierarchical Safety Assessment” approach (HSA) that is able to quantitatively analyze the violation severity of safety requirements and distinguish safer ADS configurations based on the requirements violations comparison done in a hierarchical way by following requirements importance. We apply HSA to an industrial ADS under six traffic situations. Evaluation results show that HSA is effective in distinguishing safer configurations and provides useful feedback to ADS engineers to reconfigure the ADS in a better way.

Systematic techniques to improve quality of deep neural networks (DNNs) are critical given the increasing demand for practical applications including safety-critical ones. The key challenge comes from the little controllability in updating DNNs. Retraining to fix some behavior often has a destructive impact on other behavior, causing regressions, i.e., the updated DNN fails with inputs correctly handled by the original one. This problem is crucial when engineers are required to investigate failures in intensive assurance activities for safety or trust. Search-based repair techniques for DNNs have potentials to tackle this challenge by enabling localized updates only on “responsible parameters” inside the DNN. However, the potentials have not been explored to realize sufficient controllability to suppress regressions in DNN repair tasks. In this paper, we propose a novel DNN repair method that makes use of the training history for judging which DNN parameters should be changed or not to suppress regressions. We implemented the method into a tool called Neurecover and evaluated it with three datasets. Our method outperformed the existing method by achieving often less than a quarter, even a tenth in some cases, number of regressions. Our method is especially effective when the repair requirements are tight to fix specific failure types. In such cases, our method showed stably low rates (<2 %) of regressions, which were in many cases a tenth of regressions caused by retraining.

A common approach in testing automated and autonomous driving systems (ADS) consists in running the ADS in a simulator where driving and environmental conditions are specified in terms of scenarios. An important aspect in ADS testing is to cover different driving situations in which the autonomous car must perform different types of maneuvers. In this paper, we consider the path planner of our industry partner; the path planner is responsible for deciding the path that must be followed by the autonomous car. A path is characterized by the driving characteristics (as forward acceleration, lateral acceleration, curvature, and so on) that are needed, at each time point, to implement it. For different driving characteristics, a good test suite should contain a scenario for which the path planner chooses a path that requires the application of the selected driving characteristics for a non-negligible period of time: this means that the characteristics are relevant in that path. With such a test suite, engineers can observe the different types of decision taken by the path planner, and so possibly better assess its correctness. In the paper, we introduce the notion of patterns of driving characteristics, to characterize their interaction (i.e., simultaneous or not) and measure their duration. Exploiting this definition, we propose two search-based approaches (for single and pairs of driving characteristics) to find scenarios in which such patterns occur and their duration is maximized. Experimental results show that the approaches are effective in finding scenarios for which the path planner generates paths where the different driving characteristics occur in terms of the specified pattern.

Validating Deep Neural Networks (DNNs) used for classification is of paramount importance; an approach for this consists in (i) executing the DNN over the test dataset, (ii) collecting information about classifications, and (iii) applying fault localization (FL) techniques to identify the neurons responsible for the misclassifications. DNNs can have multiple misclassification types, and so neurons responsible for one type could be different from those responsible for another type. However, depending on the granularity of the analyzed dataset, FL may not reveal these differences: failure types more frequent in the dataset may mask less frequent ones. We here propose a way to perform FL for DNNs that avoids this masking effect by selecting test data in a granular way. We conduct an empirical study, using a spectrum-based FL approach for DNNs, to assess how FL results change by changing the granularity of the analyzed test data. Namely, we perform FL by using test data with two different granularities: following a state-of-the-art approach that considers all misclassifications for a given class together, and the proposed fine-grained approach. Results show that FL should be done for each misclassification, such that practitioners have a more detailed analysis of the DNN faults and can make a more informed decision on what to fix in the DNN.

Decision systems such as Multiple-Criteria Decision Analysis systems formulate a decision process in terms of a mathematical function that takes into consideration different aspects of a problem. Testing such systems is crucial, as they are usually employed in safety-critical systems. A good test suite for these systems should be able to exercise all the possible types of decisions that can be taken by the system. Classic structural coverage criteria do not provide good test suites in this sense, as they can be fulfilled by simple tests that only cover one possible type of decision. Thus, in this paper we discuss the need for tailored coverage criteria for this class of systems, and we propose a criterion based on the perturbation of the decision systems’ parameters. We demonstrate the effectiveness of the criterion, compared to classic structural coverage criteria, on a path planner system for autonomous driving. We also discuss other benefits, such as the criterion helping explain why a decision was made during a test.

Autonomous Driving Systems (ADSs) are complex systems that must satisfy multiple requirements such as safety, compliance to traffic rules, and comfortableness. However, satisfying all these requirements may not always be possible due to emerging environmental conditions. Therefore, the ADSs may have to make trade-offs among multiple requirements during the ongoing operation, resulting in one or more requirements violations. For ADS engineers, it is highly important to know which combinations of requirements violations may occur, as different combinations can expose different types of failures. However, there is currently no testing approach that can generate scenarios to expose different combinations of requirements violations. To address this issue, in this paper, we introduce the notion of requirements violation pattern to characterize a specific combination of requirements violations. Based on this notion, we propose a testing approach named EMOOD that can effectively generate test scenarios to expose as many requirements violation patterns as possible. EMOOD uses a prioritization technique to sort all possible patterns to search for, from the most to the least critical ones. Then, EMOOD iteratively includes an evolutionary many-objective optimization algorithm to find different combinations of requirements violations. In each iteration, the targeted pattern is determined by a dynamic prioritization technique to give preferences to those patterns with higher criticality and higher likelihood to occur. We apply EMOOD to an industrial ADS under two common traffic situations. Evaluation results show that EMOOD outperforms three baseline approaches in generating test scenarios by discovering more requirements violation patterns.

The 2nd International Workshop on Data Quality Assessment for Machine Learning (DQAML'21) is organized in conjunction with the Special Interest Group on Knowledge Discovery and Data Mining (SIGKDD). This workshop aims to serve as a forum for the presentation of research related to data quality assessment and remediation in AI/ML pipeline. Data quality is a critical issue in the data preparation phase and involves numerous challenging problems related to detection, remediation, visualization and evaluation of data issues. The workshop aims to provide a platform to researchers and practitioners to discuss such challenges across different modalities of data like structured, time series, text and graphical. The aim is to attract perspectives from both industrial and academic circles.

Abstract The decentralised railway signalling systems have a potential to increase capacity, availability and reduce maintenance costs of railway networks. However, given the safety-critical nature of railway signalling and the complexity of novel distributed signalling solutions, their safety should be guaranteed by using thorough system validation methods. To achieve such a high-level of safety assurance of these complex signalling systems, scenario-based testing methods are far from being sufficient despite that they are still widely used in the industry. Formal verification is an alternative approach which provides a rigorous approach to verifying complex systems and has been successfully used in the railway domain. Despite the successes, little work has been done in applying formal methods for distributed railway systems. In our research we are working towards a multifaceted formal development methodology of complex railway signalling systems. The methodology is based on the Event-B modelling language which provides an expressive modelling language, a stepwise development and a proof-based model verification. In this paper, we present the application of the methodology for the development and verification of a distributed protocol for reservation of railway sections. The main challenge of this work is developing a distributed protocol which ensures safety and liveness of the distributed railway system when message delays are allowed in the model.

The oracle problem is a key issue in testing Autonomous Driving Systems (ADS): when a collision is found, it is not always clear whether the ADS is responsible for it. Our recent search-based testing approach offers a solution to this problem by defining a collision as avoidable if a differently configured ADS would have avoided it. This approach searches for both collision scenarios and the ADS configurations capable of avoiding them. However, its main problem is that the ADS configurations generated for avoiding some collisions are not suitable for preventing other ones. Therefore, it does not provide any guidance to automotive engineers for improving the safety of the ADS. To this end, we propose a new search-based approach to generate configurations of the ADS that can avoid as many different types of collisions as possible. We present two versions of the approach, which differ in the way of searching for collisions and alternative configurations. The approaches have been experimented on the path planner component of an ADS provided by our industry partner.

Autonomous Driving Systems (ADS) are complex critical systems that need to be thoroughly tested. Still, assessing the strength of tests for such systems is an open and complex problem. A central component of an ADS is the Path Planner, which is in charge of computing the trajectory of the autonomous vehicle. It bases its decisions on several aspects such as safety, traffic regulations, comfort, etc. These aspects can be linked to weights in a weighted cost function that ranks potential trajectories to be followed. Weight coverage has been proposed as a test criterion for tests of this type of path planner. Weight coverage measures how much the different weights (and thus the aspects they are linked to) are involved in the decisions taken by the path planner in a test scenario. All weights should be involved in at least one test. Although weight coverage has shown to be a reasonable criterion, it does not provide a clear way to drive the generation of new scenarios. In this paper, we propose a search-based approach for generating scenarios for achieving weight coverage. We introduce two variants of the approach; the first one tries to generate a scenario covering a given single weight, while the second one tries to generate scenarios covering as many weights as possible at the same time. We experimented with these approaches using the path planner provided by our industry partner, and we show that they are able to generate scenarios that cover all the weights.

Automated and autonomous driving systems (ADS) are a transformational technology in the mobility sector. Current practice for testing ADS uses virtual tests in computer simulations; search-based approaches are used to find particularly dangerous situations, possibly collisions. However, when a collision is found, it is not always easy to automatically assess whether the ADS should have been able to avoid it, without relying on offline analyses by domain experts. In this paper, we propose a definition of avoidable collision that does not rely on any domain knowledge, but only on the fact that it is possible to reconFigure the ADS (in our case, the path planner component provided by our industry partner) in a way that the collision is avoided. Based on this definition, we propose two search-based approaches for finding avoidable collisions. The first one (named sequential approach), based on current industrial practice, first searches for a collision, and then searches for an alternative configuration of the ADS which avoids it. The second one (named combined approach), instead, searches at the same time for the collision and for the alternative configuration which avoids it. Experiments show that the combined approach finds more avoidable collisions, even when the sequential approach doesn't find any; indeed, the sequential approach, in the first search, may find too severe collisions for which there is no alternative configuration that can avoid them.

Typical approaches to test Autonomous Driving Systems (ADS) generate tests in a simulation environment. A common goal in ADS testing is to find scenarios in which the car collides, as these could witness ADS faults. Recent approaches not only find a collision, but they also show whether it could be avoided: they search for a different ADS configuration (i.e., the setting of some parameters) using which the car does not collide. However, such techniques do not explain why the collision occurs and why the alternative configuration is able to avoid it. In this paper, we propose an approach to investigate the relationship between the ADS configurations and the obtained safety during driving. We first use a technique based on fuzzification to partition ADS parameters in different categories, and a spectra- based analysis to identify which categories relate to hazard and safety. Then, we consider collision scenarios by inspecting how the different ADS configurations affect the driving characteristics (e.g., acceleration and curvature) and, so, cause or avoid a collision. We applied the approach to the path planner of our industry partner, by considering three traffic situations. We observed that the path planner, to guarantee safety, should be configured differently in different situations.

Although automated machine learning (AutoML) is receiving attention in the field of data science, most AutoML open source software focuses on supervised learning tasks, and little attention has been given to unsupervised learning tasks. Moreover, AutoML has a disadvantage in that it tends to deprive users of a chance to acquire knowledge about machine learning and data science because it usually works as a black box. The purpose of this study is to help inexperienced data scientists and machine learning engineers conduct clustering data analysis, which is an unsupervised learning task, while simultaneously enabling them to acquire knowledge about clustering data analysis by extending our existing AutoML software, the machine learning support system (MALSS). The MALSS helps with clustering data analysis by automatically determining the optimal number of clusters, which is one of the main purposes of clustering data analysis. Furthermore, the MALSS helps users to acquire knowledge about clustering data analysis by generating a report after automated clustering data analysis is conducted. We validated the effectiveness of our approach by using open datasets and by running an experiment on a crowdsourcing platform.

Requirements engineering for machine learning (ML)-based systems involves unique difficulties. The core cause is the intrinsic uncertainty or unpredictability, not only in requirements and environments but also in implementation. In this paper, we discuss the impact of this type of uncertainty on requirements engineering methods such as goal-oriented requirements analysis (GORE). Many aspects in requirements analysis or prior decision making remain as hypotheses, which may be validated or invalidated with evidences from Proof-of Concept experiments, field tests, and operation. To deal with this point, we present principles of evidence-driven requirements engineering and instantiate them into a method that links GORE and ML operation (GORE-MLOps).

Bridging the gap in software development from idea to need remains a difficult task for startups, especially when users are unreachable within the black boxes of large corporations. The lack of customer collaboration results in the failure of requirements elicitation and subsequently the failure of software products. In this work, we describe a simple but effective method to enable startups to reach relevant hidden stakeholders within large corporations and elicit requirements from them. AdvisorNet leverages professional networks and social media to find senior domain experts. These carefully selected advisors bridge the gap from the outside world to the hidden corporate structure and social networks in the target corporation and enable highly successful elicitation from stakeholders. We demonstrate AdvisorNet with a real-world case study in which a London-based startup successfully connected with multiple advisors who then introduced previously unreachable users for requirements elicitation and decision-makers to achieve their goal of trialing their product within the large corporations. Finally, we provide suggestions for future research to formalize the method further and enable it to become rigorous and repeatable.

To date, although machine learning has been successful in various practical applications, generic methods of testing machine learning code have not been established yet. Here we present a new approach to test machine learning code using the possible input region obtained as a polyhedron. If an ML system generates different output for multiple input in the polyhedron, it is ensured that there exists a bug in the code. This property is known as one of theoretical fundamentals in statistical inference, for example, sparse regression models such as the lasso, and a wide range of machine learning algorithms satisfy this polyhedral condition, to which our testing procedure can be applied. We empirically show that the existence of bugs in lasso code can be effectively detected by our method in the mutation testing framework.

Significant effort is being put into developing industrial applications for artificial intelligence (AI), especially those using machine learning (ML) techniques. Despite the intensive support for building ML applications, there are still challenges when it comes to evaluating, assuring, and improving the quality or dependability. The difficulty stems from the unique nature of ML, namely, system behavior is derived from training data not from logical design by human engineers. This leads to black-box and intrinsically imperfect implementations that invalidate many principles and techniques in traditional software engineering. In light of this situation, the Japanese industry has jointly worked on a set of guidelines for the quality assurance of AI systems (in the Consortium of Quality Assurance for AI-based Products and Services) from the viewpoint of traditional quality-assurance engineers and test engineers. We report on the second version of these guidelines, which cover a list of quality evaluation aspects, catalogue of current state-of-the-art techniques, and domain-specific discussions in five representative domains. The guidelines provide significant insights for engineers in terms of methodologies and designs for tests driven by application-specific requirements.

Autonomous cars are subjected to several different kind of inputs (other cars, road structure, etc.) and, therefore, testing the car under all possible conditions is impossible. To tackle this problem, scenario-based testing for automated driving defines categories of different scenarios that should be covered. Although this kind of coverage is a necessary condition, it still does not guarantee that any possible behaviour of the autonomous car is tested. In this paper, we consider the path planner of an autonomous car that decides, at each timestep, the short-term path to follow in the next few seconds; such decision is done by using a weighted cost function that considers different aspects (safety, comfort, etc.). In order to assess whether all the possible decisions that can be taken by the path planner are covered by a given test suite T, we propose a mutation-based approach that mutates the weights of the cost function and then checks if at least one scenario of T kills the mutant. Preliminary experiments on a manually designed test suite show that some weights are easier to cover as they consider aspects that more likely occur in a scenario, and that more complicated scenarios (that generate more complex paths) are those that allow to cover more weights.

Safety assurance for automotive products is crucial and challenging. It becomes even more difficult when the variability in automotive products is considered. Recently, the notion of automotive multi-product lines (multi-PL) is proposed as a unified framework to accommodate different sources of variability in automotive products. In the context of automotive multi-PL, we propose a stability analysis for safety, motivated by our industrial collaboration, where we observed that under certain operation scenarios, safety varies drastically with small fluctuations in production parameters, environmental conditions, or driving inputs. To characterize instability, we formulate a multi-objective optimization problem, and solve it with a search-based approach. The proposed technique is applied to an industrial automotive multi-PL, and experimental results show its effectiveness to spot instability. Moreover, based on information gathered during the search, we provide some insights on both testing and quality engineering of automotive products.

Safety assessment of automotive systems is highly demanded, as failure of such systems can lead to dramatic consequences. Usually, these systems are affected by some variability as they contain some production parameters (e.g., the car power, or the braking force) that may drastically affect the behaviour of the system, and so the safety guarantees. Moreover, these systems operate in diverse environmental conditions (e.g., dry or slippery road) that may also affect the system behaviour (we name them as environmental parameters). Classical verification/validation techniques perform safety assessment by considering one particular instance of the system in one particular environmental setting. However, they do not assess the influence of system variability on the final safety. In this paper, we propose a framework for assessing the relation of production and environmental parameters with the overall safety. We first propose an approach based on simulation that assigns hazard degrees to partitions of each parameter domain (defined in terms of fuzzy sets). However, the safety could be affected by interactions of different parameters. Therefore, we also propose a clustering approach that aims at identifying patterns of parameter values providing similar hazard degrees. The approaches have been experimented on an industrial case study related to an automotive collision avoidance system implemented in Simulink. Critical parameters and parameter patterns related to potential collisions were identified and explained.

Verification of cyber-physical systems is challenging with the continuous dynamics of increasing complexity. Falsification has been considered as a promising pragmatic approach by using optimization techniques to search for input signals that lead to violation of a quantified formal specification. However, current falsification methods run a search every time from scratch given any variation in the target model. We propose a learning-based method that builds a falsifier for a family of models with a preliminary learning process. We adapt reinforcement learning techniques for this purpose. We evaluated the performance of the proposed method with a major falsification tool.

Abstract Event-B has been attracting much interest because it supports a flexible refinement mechanism that reduces the complexity of constructing and verifying models of complicated target systems by taking into account multiple abstraction layers of the models. Although most previous studies on Event-B focused on model construction, the constructed models need to be maintained. Moreover, parts of existing models are often reused to construct other models. In this paper, a method is introduced that improves the maintainability and reusability of existing Event-B models. It automatically reconstructs the refinement structure of existing models by constructing models about different sets of variables than that used in the original models, while maintaining the consistencies checked in the original models. The method automatically decomposes each refinement step into multiple steps by taking certain predicates from existing models and deriving additional predicates from the consistency conditions of existing models to create new models consistent with the original ones. By combining the decomposing of refinement steps with the composing of refinement steps, this method automatically restructures a refinement step in accordance with given sets of variables to be taken into account in refinement steps of the refactored models. The results of case studies in which large refinement steps in existing models were decomposed and existing models were restructured to extract reusable parts for constructing other models demonstrated that the proposed method facilitates effective use of the refinement mechanism of Event-B.

Energy efficiency has become one of the major concerns for today's cloud datacenters. Dynamic virtual machine (VM) consolidation is a promising approach for improving the resource utilization and energy efficiency of datacenters. However, the live migration technology that VM consolidation relies on is costly in itself, and this migration cost is usually heterogeneous as well as the datacenter. This paper investigates the following bi-objective optimization problem: how to pay limited migration costs to save as much energy as possible via dynamic VM consolidation in a heterogeneous cloud datacenter. To capture these two conflicting objectives, a consolidation score function is designed for an overall evaluation on the basis of a migration cost estimation method and an upper bound estimation method for maximal saved power. To optimize the consolidation score, a greedy heuristic and a swap operation are introduced, and an improved grouping genetic algorithm (IGGA) based on them is proposed. Lastly, empirical studies are performed, and the evaluation results show that IGGA outperforms existing VM consolidation methods.

Event-B requires engineers to satisfy proof obligations and inherit all predicates from abstract models while constructing concrete ones. Engineers typically derive predicates from abstract models through transformation with the intention of gradually refining the models. These kinds of intentions for refinement are essential for other engineers to understand the refinements. However, these are implicit and not directly specified in the models. Therefore, it is difficult to understand how each predicate in concrete models is obtained from the predicates in abstract models. This paper proposes an effective method of extracting these relationships. Our approach uses heuristics to avoid exhaustive matching between predicates. It tries to find a set of related predicates by tracing elements, such as variables and constants in predicates, and excluding predicates that use common variables, but differently. Our method facilitates understanding of the intentions of refinements and can be used to help reverse engineering by clarifying the relationships between predicates through refinements.

Nowadays it is becoming more and more attractive to execute workflow applications in the cloud because it enables workflow applications to use computing resources on demand. Meanwhile, it also challenges traditional workflow scheduling algorithms that only concentrate on optimizing the execution time. This paper investigates how to minimize execution cost of a workflow in clouds under a deadline constraint and proposes a metaheuristic algorithm L-ACO as well as a simple heuristic ProLiS. ProLiS distributes the deadline to each task, proportionally to a novel definition of probabilistic upward rank, and follows a two-step list scheduling methodology: rank tasks and sequentially allocates each task a service which meets the sub-deadline and minimizes the cost. L-ACO employs ant colony optimization to carry out deadline-constrained cost optimization: the ant constructs an ordered task list according to the pheromone trail and probabilistic upward rank, and uses the same deadline distribution and service selection methods as ProLiS to build solutions. Moreover, the deadline is relaxed to guide the search of L-ACO towards constrained optimization. Experimental results show that compared with traditional algorithms, the performance of ProLiS is very competitive and L-ACO performs the best in terms of execution costs and success ratios of meeting deadlines.

Natural ecosystems exhibit complex dynamics of interacting species. Man-made ecosystems exhibit similar dynamics and, in the case of mobile app stores, can be said to perform optimization as developers seek to maximize app downloads. This work aims to understand stability and instability within app store dynamics and how it affects fitness. The investigation is carried out with AppEco, a model of the iOS App Store, which was extended for this paper and updated to model the store from 2008 to 2014. AppEco models apps containing features, developers who build the apps, users who download apps according to their preferences, and an app store that presents apps to the users. It also models developers who use commonly observed strategies to build their apps: innovator, milker, optimizer, copycat, and flexible (the ability to choose any strategy). Results show that despite the success of the copycat strategy, there is a clear stable state for low proportion of copycats in developer populations, mirroring results in theoretical biology for producer-scrounger systems. The results also show that the best fitness is achieved when the evolutionary optimizer (as producer) and copycat (as scrounger) strategies coexist together in stable proportions.

Service composition provides a means of customized and flexible integration of service functionalities. Quality-of-service (QoS) optimization algorithms select services to adapt workflows to the non-functional requirements of the user. With increasing number of services in a workflow, previous approaches fail to achieve a sufficient reliability. Moreover, expensive ad-hoc replanning is required to deal with service failures. The major problem with such sequential application of planning and replanning is that it ignores the potential costs during the initial planning and they consequently are hidden from the decision maker. Our idea to overcome this problem is to compute a QoS optimized selection of service clusters that includes a sufficient number of backup services for each service employed. These backup services should be sufficiently distributed to prevent a task failure in case of, e.g., a network failure. To support the decision maker in the selection task, our multi-objective approach considers the possible repair costs directly in the initial composition. Our graphical user interface visualizes the resulting QoS of the workflow and the location of the services to enable the decision maker to select compositions in line with risk preferences. We prove the benefits of our approach in our detailed evaluation.

Quality of service (QoS)-aware optimal service composition aims to maximize the overall QoS value of the resulting composite service instance while meeting user-specified global QoS constraints. Traditional methods only consider as candidates service instances that implement one abstract service in the composite service and neglect those that could perform multiple abstract services. To overcome this shortcoming, this paper proposes the concept of generalized component services (GCSs), which is defined in a semantic manner, to expand the selection scope so as to achieve a better solution. A QoS-aware multigranularity service composition model is formulated and how to identify all the GCSs for a composite service is elaborated. A backtracking-based algorithm and an extended genetic algorithm are proposed to optimize the resulting composite service instance. Lastly, evaluation results of these algorithms are described.

Feedback-directed random test generation is a widely used technique to generate random method sequences. It leverages feedback to guide generation. However, the validity of feedback guidance has not been challenged yet. In this paper, we investigate the characteristics of feedback-directed random test generation and propose a method that exploits the obtained knowledge that excessive feedback limits the diversity of tests. First, we show that the feedback loop of feedback-directed generation algorithm is a positive feedback loop and amplifies the bias that emerges in the candidate value pool. This over-directs the generation and limits the diversity of generated tests. Thus, limiting the amount of feedback can improve diversity and effectiveness of generated tests. Second, we propose a method named feedback-controlled random test generation, which aggressively controls the feedback in order to promote diversity of generated tests. Experiments on eight different, real-world application libraries indicate that our method increases branch coverage by 78% to 204% over the original feedback-directed algorithm on large-scale utility libraries.

Mobile applications (apps) are software developed for use on mobile devices and made available through app stores. App stores are highly competitive markets where developers need to cater to a large number of users spanning multiple countries. This work hypothesizes that there exist country differences in mobile app user behavior and conducts one of the largest surveys to date of app users across the world, in order to identify the precise nature of those differences. The survey investigated user adoption of the app store concept, app needs, and rationale for selecting or abandoning an app. We collected data from more than 15 countries, including USA, China, Japan, Germany, France, Brazil, United Kingdom, Italy, Russia, India, Canada, Spain, Australia, Mexico, and South Korea. Analysis of data provided by 4,824 participants showed significant differences in app user behaviors across countries, for example users from USA are more likely to download medical apps, users from the United Kingdom and Canada are more likely to be influenced by price, users from Japan and Australia are less likely to rate apps. Analysis of the results revealed new challenges to market-driven software engineering related to packaging requirements, feature space, quality expectations, app store dependency, price sensitivity, and ecosystem effect.

Method names are important for the software development process. It has been shown by some studies that the quality of method names affects software comprehension. In response, some approaches that evaluate comprehensibility of method names have been proposed. However, the effectiveness of existing approaches is limited because they focus on part of names.

Service-Oriented Computing enables the composition of loosely coupled services provided with varying Quality of Service (QoS) levels. Selecting a near-optimal set of services for a composition in terms of QoS is crucial when many functionally equivalent services are available. As the number of distributed services, particularly in the cloud, is rising rapidly, the impact of the network on the QoS keeps increasing. Despite this, current approaches do not differentiate between the QoS of services themselves and the network. Therefore, the computed latency differs from the actual latency, resulting in suboptimal QoS. Thus, we propose a network-aware approach that handles the QoS of services and the QoS of the network independently. First, we build a network model in order to estimate the network latency between arbitrary services and potential users. Our selection algorithm then leverages this model to find compositions with a low latency for a given execution policy. We employ a self-adaptive genetic algorithm which balances the optimization of latency and other QoS as needed and improves the convergence speed. In our evaluation, we show that our approach works under realistic network conditions, efficiently computing compositions with much lower latency and otherwise equivalent QoS compared to current approaches.

In systems that require several services to collaborate, specifying coordination protocols is vital, but costly. Additionally, several properties, which are derived from laws, regulations, requirements, etc., must be satisfied. Coordination protocol composition approaches construct specific protocols in a cost effective manner in accordance with the composition intentions. However, existing composition approaches are insufficient in terms of satisfying properties. Existing approaches use concrete specifications to identify composition intentions and do not consider interference between compositions. Herein we propose a new composition approach in which a developer directly expresses his or her intentions as constraints via metadata, and then the system searches for optimal composition methods based on the constraints.

Globally distributed software development teams face problems with software development life cycle phases, as the distributed nature of each of these phases make it even more challenging to communicate between the stakeholders. Global distance can give rise to incomplete requirements handovers which make the situation more difficult. It is important to address this issue as the end product is likely to deliver less business value when such problems arise. In this research, we propose a process to facilitate non-verbal communication among globally distributed requirements engineering teams. The focus of this research is the situation that occurs after requirements are handed to another site. Our proposed process endeavors to ensure that incomplete and conflicting requirements are identified and mitigated.

Service composition algorithms are used for realizing loosely coupled interactions in Service-Oriented Computing. Starting from an abstract workflow, concrete services are matched, based on their QoS, with the preferences and constraints of users. Current approaches usually only consider static QoS values and find a single solution consisting of one concrete service for each workflow task. In a business-to-business (B2B) environment, though, there are additional requirements for service compositions: 1) a high number of invocations, and 2) a high reliability. Thus, we introduce a probabilistic approach on the basis of a new QoS model to solve the composition problem for such long-term B2B service compositions. For each task and for every point in time, we determine the most appropriate services and backup services for a specific user. Thus, the selection depends on the actual response time and reliability, or recent invocation failures or timeouts. For that purpose, we propose an adaptive genetic algorithm that employs our QoS model and determines backup services dynamically based on the required reliability. Our evaluations show that our approach significantly increases the utility of long-term compositions compared with standard approaches in the envisioned B2B environments.

The concept of trust in web services mainly deals with the degree of belief that a client or a group of clients have over services functioning satisfactorily and providing the expected results. With services being invoked in composition with each other, computing the trust of the composition and selecting services that deliver the highest trust for it becomes a desired goal. In this paper we demonstrate how using Bayesian networks and its supporting queries, we can select the set of services among all candidates that would provide highest global trust.

Optimizing the Quality-of-Service (QoS) levels of a service workflow is essential for the user satisfaction in Service-oriented Computing. For that purpose, QoS computation models are applied to reflect the actual QoS experienced by the user during service execution. Current QoS models ignore the possible dependencies of QoS attributes, such as the dependency on the time of the execution or on the input data supplied to the service. Apart from that, composition approaches consider only single workflows during service selection, narrowing the number of possible compositions. Thus, we introduce a novel QoS model that covers QoS dependencies and discuss how this model can be used to consider multiple workflows at the same time. Moreover, we adopt a multi-objective optimization approach to offer solutions varying in QoS such as finishing time and price, allowing the user to make fine-grained decisions.

The technology for web services has facilitated composition of a new service by combining existing services. The resulting composite service is required to guarantee Quality of Service (QoS), such as price, in addition to the demanded function. Each composite service faces computationally-expensive service selection by exploring enormous service combinations for optimization of a utility function and satisfaction of global constraints. In addition, each concrete service has difficulties in determining QoS that is profitable while attractive for customers, or composite services. Specifically, information on rival services is generally secret and changeable, which is required for the optimal decision. In response to these problems, this paper proposes an algorithm for service selection using Vickrey auction. In the first phase of the proposed algorithm, the composite service selects some best concrete services through the reverse Vickrey auction, only considering the utility function. In the second phase, the global constraints are satisfied through adjustments based on the Vickrey auction. The proposed algorithm avoids full exploration of services combinations for efficient computation, while encouraging providers to declare their best QoS without caring about extra profit.

Service-Oriented Computing (SOC) enables the composition of loosely coupled services provided with varying Quality of Service (QoS) levels. Selecting a (near-)optimal set of services for a composition in terms of QoS is crucial when many functionally equivalent services are available. With the advent of Cloud Computing, both the number of such services and their distribution across the network are rising rapidly, increasing the impact of the network on the QoS of such compositions. Despite this, current approaches do not differentiate between the QoS of services themselves and the QoS of the network. Therefore, the computed latency differs substantially from the actual latency, resulting in suboptimal QoS for service compositions in the cloud. Thus, we propose a network-aware approach that handles the QoS of services and the QoS of the network independently. First, we build a network model in order to estimate the network latency between arbitrary services and potential users. Our selection algorithm then leverages this model to find compositions that will result in a low latency given an employed execution policy. In our evaluation, we show that our approach efficiently computes compositions with much lower latency than current approaches.

Web service composition provides a means of customized and flexible integration of service functionalities. Quality-of-Service (QoS) optimization algorithms select services in order to adapt workflows to the non-functional requirements of the user. With increasing number of services in a workflow, previous approaches fail to achieve a sufficient reliability. Moreover, expensive ad-hoc replanning is required to deal with service failures. The major problem with such sequential application of planning and replanning is that it ignores the potential costs during the initial planning and they consequently are hidden from the decision maker. Our basic idea to overcome this substantial problem is to compute a QoS optimized selection of service clusters that includes a sufficient number of backup services for each service employed. To support the human decision maker in the service selection task, our approach considers the possible repair costs directly in the initial composition. On the basis of a multi-objective approach and using a suitable service selection interface, the decision maker can select compositions in line with his/her personal risk preferences.

Due to the increase in Web services, many recent studies have been addressing the service selection problem based on non-functional or quality aspects. Our study incorporates combinational use of functionally-equivalent services into the problem to compose an application of higher quality or with additional value. However, when such combinational use is introduced, computational cost for the service selection becomes much higher. In this work, we propose a set of methods that reduce the additional cost for the QoS (Quality of Service)-based service selection considering combinational use. This approach achieves low cost by considering only effective combinations. The experimental results show that it can reduce computational cost regardless of the number of services and whatever their QoS values are while keeping the effectiveness of combinational use.

Service-Oriented Architecture enables the composition of loosely coupled services provided with varying Quality of Service (QoS) levels. Given a composition, finding the set of services that optimizes some QoS attributes under given QoS constraints has been shown to be NP-hard. Therefore, heuristic algorithms are widely used, finding acceptable solutions in polynomial time. Still the time complexity of such algorithms can be prohibitive for real-time use, especially if the algorithms are required to run until they find near-optimal solutions. Thus, we propose a heuristic approach based on Hill-Climbing that makes effective use of an initial bias computed with Linear Programming, and works on a reduced search space. In our evaluation, we show that our approach finds near-optimal solutions and achieves a low time complexity.

With the steadily growing number of service providers the competition becomes more and more intense. In order to find a distinctive edge over other competitors, automatic service composition can be applied to further adapt to the requirements of the users. Most of the current composition approaches can be categorized as either planning or selection algorithms. The former automatically create workflows and tend to neglect Quality-of-Service (QoS) concerns, drawing compositions useless if user constraints are violated. Selection algorithms on the other hand optimize the QoS but provide no functional flexibility. In this paper we show how the strengths of both approaches can be combined by leveraging common characteristics of service registries. Therefore we utilize a data structure which arranges functionally similar services in clusters and computes the QoS of each cluster. Our planning tool composes workflows consisting of these clusters, taking the QoS of the clusters into account. This way, the utility in general and the reliability of the composed workflows are significantly increased. We prove the feasibility and the benefits of our functional clustering in our detailed evaluation.

The correctness of business process models is critical for IT system development. The properties of business processes need to be analyzed when they are designed. In particular, business processes usually have various constraints on time and resources, which may cause serious problems like bottlenecks and deadlocks. In this paper, we propose an approach based on the model checking technique for verifying business process models with temporal and resource constraints. First, we extend Business Process Modeling Notation (BPMN) to handle these constraints. Then, we provide a mapping of the business process models described with this extended BPMN onto timed automata that can be verified by the UPPAAL model checker. This approach helps to eliminate various problems with time and resources in the early phase of development, and enables the quality assurance of business process models.

Geographic information systems use databases to map keywords to places. These databases are currently most often created by using a top-down approach based on the geographic definitions. However, there is a problem with this approach in that these databases only contain location definitions such as addresses and place names, which does not allow for searches using keywords other than these words. Additionally, they do not give any information on the popularity, e.g., which is more popular among the places indexed by the same keyword. A bottom-up approach, based on the actual usage of words, can address these problems. We propose a method to aggregate tagging data and extract places related to a tag using the pair of a tag and a geo-tagged photo. We target the co-occurrence of a tag and the geolocation and represent the places related to a tag as a probability distribution over the longitudes and latitudes. We applied our method to data on the photo sharing service Flickr and experimentally confirmed that our method made it possible to highly-accurately extract places related to tags. Our direct bottom-up approach enables the extraction of place information that is not obtained by using traditional top-down approaches.

Event-driven services, or notification and event-handling services, provide specific information or functionality, repeatedly in a periodical or reactive way in response to events in which consumers are interested. Although such services have played primary roles in distributed computing paradigms and human activities, machine-accessible description or matching methods for such services has not been investigated. This paper presents and discusses foundations for description and matching of triggering capabilities in event-driven services. Specifically, requirement description by the consumer side and service description by the provider side are presented as well as their matching, including specific concepts such as configurability by the consumer side. This paper provides discussion on both of a foundational, general theory and a prototype implementation.

The Business Process Execution Language (BPEL) has become the standard in Web service composition, so that been adopted by industries to create mission critical services. The business process reliability especially business process consistency when some services failed is critical to composite services especially involving various organizations. However, it is difficult to conduct business process reliability testing for composite services due to complex business process in service composition, remote deployed services and long time testing duration. Little research is done to test fault tolerance capabilities of composite services only from low level. This paper proposes a technique on how to test the reliability of composite service defined in BPEL from the view of business semantics with little cost using fault injection. We present an approach for reliability testing of web services by using service stubs with semantic faults instead of real services which can be placed at service provider side or service consumer side. Dislike existing studies, we focus on how to find the reliability problems associating with business process - called semantics as the problems are not pure coding error but faults related to business process. In addition, the behavior of composite services in BPEL is analyzed when there are faults in orchestrated services invoked. Finally, a case study is given to show the whole process of reliability testing for composite web service.

Formal methods allow to verify several properties of specifications and implementations. Intra-specification consistency means that a specification does not contradict itself. When specifications evolve over time, one also wants to check inter-specification consistencies, which mean that specifications defined earlier in the development cycle also hold at a later point in time. VDM++ is a popular and easy-to-use formal specification language. It uses testing instead of formal proofs to validate the consistency of specifications. The strictness of validations thus depends on the completeness of the corresponding test suites. Unfortunately, VDM++ does not support the verification of inter-specification consistencies. We define VDM-R, an extension of VDM++, which allows to annotate relationships between specifications. We also provide the tool VR2EvtB to translate from VDM-R to Event-B. Using an Event-B verifier, we can then formally validate intra- and inter-specification consistencies in an almost fully-automated process.

In e-business, agents need to coordinate with each other. Coordination protocols that specify the defining orders of message passing are very important. The scale of e-Business grows with the advancement of technology, and the number of agents involved continues to increase. Specifying the coordination protocols for so many participating agents is a complex task. A coordination protocol composition approach reduces the complexity of specifying such a coordination protocol. It treats coordination protocols as individual parts and composes them to construct the intended protocols. However, existing approaches do not sufficiently reduce the complexity when a coordination protocol is composed several times, because too many configurations are required to specify a composition. A new approach is proposed that uses metadata to specify the compositions by specifying only one configuration. It can reduce the number of configurations when a coordination protocol is composed several times.

Multi-agent systems need protocols to coordinate among agents implemented by different owners. However, specifying coordination protocols for many participating agents is a complex task. A protocol composition approach, which can reduce the complexity of specifying such a coordination protocol, must specify how to compose coordination protocols in a composition configuration. Current protocol compositions cannot sufficiently reduce the complexity because composing a protocol several times requires the specification of too many configurations. We propose a protocol composition approach that can specify composition configurations in an abstract way. We assign metadata to the messages in a coordination protocol and use the metadata to specify the configuration. An abstract configuration using metadata can be applied to various protocol compositions and can reduce the number of specifications required for configuration composition.

BPEL and similar languages have been provided a foundation for process-based implementation of composite Web services. These languages allow definition of orchestration processes without concrete binding information of involved partners. Although this approach facilitates customization of service discovery and selection before process execution, partner management during process execution has not been considered, such as rebinding of service providers upon events. In response to this problem, this study proposes a description model that facilitates runtime partner management in process-based services. The proposed model includes policy descriptions that allow insertion and customization of binding behavior according to surrounding environments and user preferences. The model also includes extensions in the standard process notation in order to allow process developers to give constraints to avoid adoption of inadequate policies as well as to define handlers to have common rebinding behavior reused. The proposed descriptions have been implemented and used in an agent framework for multimedia services.

This paper addresses the issue of coordination mechanism selection in agent based service oriented computing systems. We aim to strengthen the current research on service selection and composition with a dynamic choice of the underlying coordination mechanisms. Indeed, since the environment in such systems might dynamically change, and the agents can have several protocols at their disposal, they need to select the most adequate one for service provision/request upon the prevailing circumstances. In this paper, we present a joint protocol selection mechanism, which enables agents to decide together the protocols they will use to coordinate each other during a service provision.

Negotiation between robots is often needed in wireless ad-hoc robot network. In dynamic coverage problem, wireless ad-hoc robots move in the field negotiating with other robots frequently. Robots are tied with P2P network and determine their actions autonomously by negotiations with other robots. The topic of this paper is a negotiation for multiagent system (MAS) in the problem of robots activity in dynamic coverage, which is the problem of robots to cover all the areas of a free space by their sensors in the shortest possible time. This paper introduce a novel algorithm for robots' negotiation for MAS in the dynamic coverage problem by using PID-filter. This algorithm reduced useless motion of robots and decreased the time concerning the convergence of a robot's action for dynamic coverage problem. The claim made in this paper for new negotiation algorithm for MAS are supported by comprehensive experimental results and discusses future research directions

Web services technology for interoperability, dynamic discovery and integration of distributed components is to be applied in various environments, such as networks of relatively narrow and unstable wireless connections. However, in such environments it is necessary to deal with resource limitation. Our work adopts mobile agent technology in response to this problem and presents a mobile agent framework for Web Services integration. This framework utilizes BPEL (Business Process Execution Language for Web Services) for description of the integration logic, and provides declarative descriptions for physical behaviors (migration and cloning). This facilitates addition or change of physical behaviors according to the environmental conditions while preserving the integration logic. This paper especially focuses on formal definition and discussion of our framework.