High-quality datasets are essential for machine learning-based intrusion detection systems, which are considered a promising defense for cyber-physical systems against Advanced Persistent Threats (APTs). However, existing datasets often are not built to capture the long, multi-stage nature of real APT campaigns, and they are labeled as general cyber-attacks rather than explicitly as APTs. To address this gap, we propose a methodology for creating a semi-synthetic, labeled dataset that reflects the complex attack paths typical of APTs targeting cyber-physical environments. Our approach integrates realistic network traffic gathered from a real testbed with multi-step APT attack scenarios modeled on the well-established MITRE ATT&CK framework and CVE exploits repository. The cAPTure dataset provides a rich basis for evaluating intrusion detection systems, enabling an evaluation methodology that relates false positive rate and time-to-detection, two metrics that are crucial for practical, real-world NIDS deployment.

Abstract The ever-evolving landscape of attacks and the growing complexity of ICT systems make crafting anomaly-based intrusion and error detectors difficult: they must accurately detect attacks and promptly perform detections. Although improving and comparing the detection capability is the focus of most research works, the timeliness of the detection is less considered and often insufficiently evaluated or discussed. In this paper, we argue the relevance of measuring the temporal latency of attacks, and we propose an evaluation approach for detectors to ensure a trade-off between correct and in-time detection. Briefly, the approach relates the false positive rate to the temporal latency of attacks, ultimately leading to guidelines for configuring a detector. We discuss and apply the strategy to compose datasets for intrusion detection that can support the computation of our metrics. We exercise our approach by evaluating different intrusion and error detectors in three industrial cases: i) an embedded railway on-board system that optimizes public mobility, ii) an edge device for the Industrial Internet of Things, and iii) an IoT network that monitors an industrial facility. Results show that considering latency in addition to traditional metrics like the false positive rate, precision, and recall gives an additional fundamental perspective on the actual performance of the detector and should be considered when assessing and configuring intrusion detectors.

ABSTRACT Background Modern critical systems often require to take decisions and classify data and scenarios autonomously without having detrimental effects on people, infrastructures or the environment, ensuring desired dependability attributes. Researchers typically strive to craft classifiers with perfect accuracy, which should be always correct and as such never threaten the encompassing system. Unfortunately, this is a very unrealistic goal, as classification tasks are typically complex and may encounter a wide variety of unexpected operating conditions and unknown inputs. Methods Classifiers should be considered as building blocks that interact with other components that help rejecting those predictions that are suspected to be misclassifications, triggering system‐level mitigation strategies instead. Fail‐Controlled Classifiers (FCCs) are software components that can either correctly classify, misclassify, or reject outputs: ideally, they would reject all and only outputs that correspond to misclassifications. Nine different FCCs are presented: Self‐Checking Classifiers (SCC), Watchdog Timers (WT), Input Processor (IP), Output processor (OP), Safety Wrapper (SW), Recovery Blocks (RB), weighted and non‐weighted Voting (VT, WVT) and Stacking (STK). Results These 9 FCCs are instantiated in experiments with tabular and image classifiers, showing their potential in rejecting most misclassifications and paving the ways for trustworthy decisions to be deployed in critical systems. If the system can tolerate more omissions, the IP FCC is a good choice. On the other hand, if achieving the highest accuracy is the priority, RB FCC performs better. Conclusions Findings show that FCCs do not primarily aim at improving correct classifications, but allow for transforming many misclassifications into rejections, which may be easily handled by the encompassing system and paving the way for trustworthy decisions to be deployed in critical systems.

Abstract Machine Learning algorithms that perform classification are increasingly been adopted in Information and Communication Technology (ICT) systems and infrastructures due to their capability to profile their expected behavior and detect anomalies due to ongoing errors or intrusions. Deploying a classifier for a given system requires conducting comparison and sensitivity analyses that are time-consuming, require domain expertise, and may even not achieve satisfactory classification performance, resulting in a waste of money and time for practitioners and stakeholders. This paper predicts the expected performance of classifiers without needing to select, craft, exercise, or compare them, requiring minimal expertise and machinery. Should classification performance be predicted worse than expectations, the users could focus on improving data quality and monitoring systems instead of wasting time in exercising classifiers, saving key time and money. The prediction strategy uses scores of feature rankers, which are processed by regressors to predict metrics such as Matthews Correlation Coefficient (MCC) and Area Under ROC-Curve (AUC) for quantifying classification performance. We validate our prediction strategy through a massive experimental analysis using up to 12 feature rankers that process features from 23 public datasets, creating additional variants in the process and exercising supervised and unsupervised classifiers. Our findings show that it is possible to predict the value of performance metrics for supervised or unsupervised classifiers with a mean average error (MAE) of residuals lower than 0.1 for many classification tasks. The predictors are publicly available in a Python library whose usage is straightforward and does not require domain-specific skill or expertise.

The ever-evolving landscape of attacks, coupled with the growing complexity of ICT systems, makes crafting anomaly-based intrusion detectors and error detectors difficult: they must accurately detect attacks and promptly perform detections. Although improving and comparing the detection capability is the focus of most research works, the timeliness of the detection is less considered and often insufficiently evaluated or discussed. In this paper, we argue the relevance of measuring the temporal latency of attacks and errors, and we propose an evaluation approach for detectors to ensure a trade-off between correct and in-time detection. Briefly, the approach relates the false positive rate with the temporal latency of attacks and errors, ultimately leading to guidelines for configuring a detector. We apply our approach by evaluating different intrusion and error detectors in two industrial cases: i) an embedded railway on-board system that optimizes public mobility, and ii) an edge device for the Industrial Internet of Things. Our results show that considering latency in addition to traditional metrics like the false positive rate, precision, and coverage gives an additional fundamental perspective on the actual performance of the detector and should be considered when assessing and configuring anomaly detectors.

Domain experts are desperately looking to solve decision-making problems by designing and training Machine Learning algorithms that can perform classification with the highest possible accuracy. No matter how hard they try, classifiers will always be prone to misclassifications due to a variety of reasons that make the decision boundary unclear. This complicates the integration of classifiers into critical systems, where misclassifications could directly impact people, infrastructures, or the environment. The paper proposes to consider a classifier as a structural part of the system instead of an individual component to be tested in isolation and included in the system afterward. This allows for omitting those predictions that are suspected to be misclassifications, triggering system-level mitigation strategies. The resulting fail-controlled classifiers (FCCs) are software components that can correctly classify, misclassify, or omit outputs: ideally, they would omit all and only outputs that correspond to misclassifications. After presenting the theoretical foundations of FCCs, the paper proposes metrics to quantify their performance, 5 software architectures for FCCs, and an experimental analysis involving tabular data and image classifiers. Overall, this paper advocates the need for a system and software design in which ML classifiers are not separate components, but should rather be considered building blocks that interact with other components for improved performance.

Recent years have seen a growing involvement of researchers and practitioners in crafting Deep Neural Networks (DNNs) that seem to outperform existing machine learning approaches for solving classification problems as anomaly-based error and intrusion detection. Undoubtedly, classifiers may be very diverse among themselves, and choosing one or another is typically due to the specific task and target system. Designing and training the optimal tabular data classifier requires extensive experimentation, sensitivity analyses, big datasets, and domain-specific knowledge that may not be available at will or considered a non-strategical asset by many companies and stakeholders. This paper compares, using a total of 23 public datasets: i) traditional (tree-based, statistical) supervised classifiers, ii) DNNs that are specifically designed for classifying tabular data, iii) DNNs for image classification that are applied to tabular data after converting data points into images, alone and as ensembles. Experimental results and related discussions show clear advantages in adopting tree-based classifiers for anomaly-based error and intrusion detection in tabular data as they outperform their competitors, including DNNs. Then, individual classifiers are compared against ensembles using different combinations of the classifiers considered in this study as base-learners, providing a unified final response through many meta-learning strategies. Results show that there is no benefit in building ensembles instead of using a tree-based classifier as Random Forests, eXtreme Gradient Boosting or Extra Trees. The paper concludes that anomaly-based error and intrusion detectors for critical systems should use the old (but gold) tree-based classifiers, which are also easier to fine-tune, and understand; plus, they require less time and resources to learn their model.

Machine Learning (ML) algorithms that perform classification may predict the wrong class, experiencing misclassifications. It is well-known that misclassifications may have cascading effects on the encompassing system, possibly resulting in critical failures. This paper proposes SPROUT, a Safety wraPper thROugh ensembles of UncertainTy measures, which suspects misclassifications by computing uncertainty measures on the inputs and outputs of a black-box classifier. If a misclassification is detected, SPROUT blocks the propagation of the output of the classifier to the encompassing system. The resulting impact on safety is that SPROUT transforms erratic outputs (misclassifications) into data omission failures, which can be easily managed at the system level. SPROUT has a broad range of applications as it fits binary and multi-class classification, comprising image and tabular datasets. We experimentally show that SPROUT always identifies a huge fraction of the misclassifications of supervised classifiers, and it is able to detect all misclassifications in specific cases. SPROUT implementation contains pre-trained wrappers, it is publicly available and ready to be deployed with minimal effort.

Anomaly-based intrusion detectors are machine learners trained to distinguish between normal and anomalous data. The normal data is generally easy to collect when building the train set; instead, collecting anomalous data requires historical data or penetration testing campaigns. Unfortunately, the first is most often unavailable or unusable, and the latter is usually expensive and unfeasible, as it requires hacking the target system. It turns out that the possibility of training an intrusion detector without attack knowledge, i.e., without anomalies, is attractive. This paper reviews strategies to train anomaly detectors in the absence of anomalies, from shallow machine learning to deep learning and computer vision approaches, and applies such strategies to the domain of intrusion detection. We experimentally show that training an intrusion detector without attack knowledge is effective when normal and attack data distributions are distinguishable. Detection performance severely drops in the case of complex (but more realistic) datasets, making all the existing solutions inadequate for real applications. However, the recent advancements of out-of-distribution research in deep learning and computer vision show interesting prospective results.

With the growing processing power of computing systems and the increasing availability of massive datasets, machine learning algorithms have led to major breakthroughs in many different areas. This applies also to resource-constrained IoT and edge devices, which will often benefit from relatively small – but smart – local anomaly detection tasks that aim at protecting the device, or the information they convey from sensors towards a central node. This provides the device with fault detection capabilities that are typically required when engineering dependable devices, services or systems. This paper overviews a pitfall-free process to provide small devices with anomaly detection capabilities, to make them self-aware of their health condition, and possibly take appropriate countermeasures. Our methodology applies to a wide range of Linux-based devices: we show an application to a specific ARANCINO device, which has already been successfully used in many smart cities and sensing applications. We craft anomaly detectors that are very effective in detecting most of the anomalies. Additionally, we comment on the beneficial impact of time-series analysis, which could help improve detection performance even further, allowing to equip any small device with responsive and accurate anomaly detection machinery.

There is an astounding growth in the adoption of machine learners (MLs) to craft intrusion detection systems (IDSs). These IDSs model the behavior of a target system during a training phase, making them able to detect attacks at runtime. Particularly, they can detect known attacks, whose information is available during training, at the cost of a very small number of false alarms, i.e., the detector suspects attacks but no attack is actually threatening the system. However, the attacks experienced at runtime will likely differ from those learned during training and thus will be unknown to the IDS. Consequently, the ability to detect unknown attacks becomes a relevant distinguishing factor for an IDS. This study aims to evaluate and quantify such ability by exercising multiple ML algorithms for IDSs. We apply 47 supervised, unsupervised, deep learning, and meta-learning algorithms in an experimental campaign embracing 11 attack datasets, and with a methodology that simulates the occurrence of unknown attacks. Detecting unknown attacks is not trivial: however, we show how unsupervised meta-learning algorithms have better detection capabilities of unknowns and may even outperform classification performance of other ML algorithms when dealing with unknown attacks.

The railway domain is regulated by rigorous safety standards to ensure that specific safety goals are met. Often, safety-critical systems rely on custom hardware-software components that are built from scratch to achieve specific functional and non-functional requirements. Instead, the (partial) usage of Commercial Off-The-Shelf (COTS) components is very attractive as it potentially allows reducing cost and time to market. Unfortunately, COTS components do not individually offer enough guarantees in terms of safety and security to be used in critical systems as they are. In such a context, RFI (Rete Ferroviaria Italiana), a major player in Europe for railway infrastructure management, aims at equipping track-side workers with COTS devices to remotely and safely interact with the existing interlocking system, drastically improving the performance of maintenance operations. This paper describes the first effort to update existing (embedded) railway systems to a more recent cyber-physical system paradigm. Our Remote Worker Dashboard (RWD) pairs the existing safe interlocking machinery alongside COTS mobile components, making cyber and physical components cooperate to provide the user with responsive, safe, and secure service. Specifically, the RWD is a SIL4 cyber-physical system to support maintenance of actuators and railways in which COTS mobile devices are safely used by track-side workers. The concept, development, implementation, verification, and validation activities to build the RWD were carried out in compliance with the applicable CENELEC standards required by certification bodies to declare compliance with specific guidelines.

RGB cameras are one of the most relevant sensors for autonomous driving applications. It is undeniable that failures of vehicle cameras may compromise the autonomous driving task, possibly leading to unsafe behaviors when images that are subsequently processed by the driving system are altered. To support the definition of safe and robust vehicle architectures and intelligent systems, in this paper we define the failure modes of a vehicle camera, together with an analysis of effects and known mitigations. Further, we build a software library for the generation of the corresponding failed images and we feed them to six object detectors for mono and stereo cameras and to the self-driving agent of an autonomous driving simulator. The resulting misbehaviors with respect to operating with clean images allow a better understanding of failures effects and the related safety risks in image-based applications.

General Purpose GPUs (GPGPUs) are highly susceptible to both transient and permanent faults. This is a serious concern for their safe and reliable usage in many domains, from autonomous driving to High Performance Computing. The research and industrial community responded fiercely to this issue, by analyzing failures impact and devising failure mitigation strategies. This led to the definition of several failure modes and mitigation approaches. Unfortunately, these are often based on different foundations, and it is not easy to position them in a consistent view. This work elaborates a GPGPU failures model, identifying relations between the GPGPU failure modes and components, and then it analyzes mitigations proposed in the literature. By proposing a unified view on failures and mitigations, the resulting model i) positions each research on the subject, ii) easily identifies the current gaps, and iii) sets the basis for further research on GPGPU failures.

Recent years have seen an astounding growth in the adoption of Machine Learning algorithms to classify data gathered through monitoring activities. Those algorithms can effectively classify data as system indicators, network packets, and logs according to a model they infer during training. This way, they provide sophisticated means to conduct intrusion detection by suspecting anomalies due to attacks in the value of those features. Additionally, Meta-Learners as Bagging and Boosting build ensembles of homogeneous classifiers that are known to improve classification performance with positive impact on intrusion detection. On the other hand, it is not yet clear if ensembles of heterogeneous or diverse classifiers can build better intrusion detectors. To such extent, we first recap on n-version programming, k-out-of-m (k-o-o-m) systems and the role of diversity. Then, we present k-o-o-m systems of classifiers for intrusion detection, expanding on meta-learning and diversity measures to be applied to classifiers. This paves the way for an experimental campaign which exercises supervised and unsupervised classifiers as well as k-o-o-m voting ensembles. After presenting and discussing results, we conclude that voting ensembles of diverse classifiers does not improve intrusion detection. Therefore, while voting has been acknowledged since decades as a staple to manage n-version programming for reliable systems engineering, it is not as effective as a meta-learner to improve classification performance of intrusion detectors.

Abstract A cyber‐physical system‐of‐systems (CPSoS) can be defined as a system‐of‐systems (SoS), composed of several operable and autonomous constituent systems (CSs) that are themselves cyber‐physical systems (CPSs). A main challenge in integrating CPSoS to function as a single integrated system is the autonomy of its components, which may result in undesirable, unsecure, or even unsafe situations. In this paper, we advocate that in order to facilitate the integration of CPSs within the overall context of their CPSoS, we may need to adjust their level of autonomy in a way that enables them to perform their activities and avoid undesirable, unsecure , and unsafe situations. Reducing such situations surely contributes to the dependability of the CPSoS. In particular, we propose a novel model‐based approach for modeling and analyzing the autonomy levels of CPSs based on their awareness concerning their operational environment as well as their capability to react in a timely, secure, and safe manner while performing their activities. The model is further described in a UML profile and applied to represent activities for autonomous driving scenarios. Using a driving simulator, we implement such models on a target vehicle, and we show the resulting safety improvement, especially in terms of reduced collisions.

Anomaly detection can infer the presence of errors without observing the target services, but detecting variations in the observable parts of the system on which the services reside. This is a promising technique in complex software-intensive systems, because either instrumenting the services' internals is exceedingly time-consuming, or encapsulation makes them not accessible. Unfortunately, in such systems anomaly detection is often ineffective due to their dynamicity, which implies changes in the services or their expected workload. Here we present our approach to enhance the efficacy of anomaly detection in complex, dynamic software-intensive systems. After discussing the related challenges, we present MADneSs, an anomaly detection framework tailored for the above systems that includes an adaptive multi-layer monitoring module. Monitored data are then processed by the anomaly detector, which adapts its parameters depending on the current system behavior. An anomaly alert is provided if the analysis conducted by the anomaly detector identify unexpected trends in the data. MADneSs is evaluated through an experimental campaign on two service-oriented architectures; software faults are injected in the application layer, and detected through monitoring of underlying system layers. Lastly, we quantitatively and qualitatively discuss our results with respect to state-of-the-art solutions, highlighting the key contributions of MADneSs.

The injection of software faults in source code requires accurate knowledge of the programming language, both to craft faults and to identify injection locations. As such, fault injection and code mutation tools are typically tailored for a specific language and have limited extensibility. In this paper we present a model-driven approach to craft and inject software faults in source code. While its concrete application is presented for Java, the workflow we propose does not depend on a specific programming language. Following Model-Driven Engineering principles, the faults and the criteria to select injection locations are described using structured, machine-readable specifications based on a domain-specific language. Then, automated transformations craft artifacts based on OCL and Java, which represent the faults to be injected and are able to select the candidate injection locations. Finally, artifacts are executed against the target source code, performing the injection in the desired locations. We devise a supporting tool and exercise the approach injecting 13 different kinds of software faults in the Java source code of six different projects.

RGB cameras are arguably one of the most relevant sensors for autonomous driving applications. It is undeniable that failures of vehicle cameras may compromise the autonomous driving task, possibly leading to unsafe behaviors when images that are subsequently processed by the driving system are altered. To support the definition of safe and robust vehicle architectures and intelligent systems, in this paper we define the failure modes of a vehicle camera, together with an analysis of effects and known mitigations. Further, we build a software library for the generation of the corresponding failed images and we feed them to the trained agent of an autonomous driving simulator: the misbehavior of the trained agent allows a better understanding of failures effects and especially of the resulting safety risk.

Online model-based testing is one of the most suitable techniques to assess the proper behavior of service orchestrations. However, the diverse panorama in terms of modeling languages and test case generation tools is a limitation to widespread adoption. We advocate that the application of Model-Driven Engineering principles as meta-modeling and model transformation can cope with this problem, improving the interoperability of artifacts in the test case generation process, thus bringing benefits in case of agile development processes, where system and technology evolution is frequent. In this paper, we present our contribution to this idea, introducing i) a reference metamodel, which stores the business process behavior and the information to generate input models for testing tools, and ii) transformations from orchestration languages towards testing tools. The proposed approach is implemented in a testing framework and evaluated on a case study where multiple orchestrations are expressed in two languages. Also, the paper presents how test cases are appropriately generated and successfully executed, starting from an orchestration model as a consequence of successful transformations.

Anomaly detection aims at finding patterns in data that do not conform to the expected behavior. It is largely adopted in intrusion detection systems, relying on unsupervised algorithms that have the potential to detect zero-day attacks; however, efficacy of algorithms varies depending on the observed system and the attacks. Selecting the algorithm that maximizes detection capability is a challenging task with no master key. This paper tackles the challenge above by devising and applying a methodology to identify relations between attack families, anomaly classes and algorithms. The implication is that an unknown attack belonging to a specific attack family is most likely to get observed by unsupervised algorithms that are particularly effective on such attack family. This paves the way to rules for the selection of algorithms based on the identification of attack families. The paper proposes and applies a methodology based on analytical and experimental investigations supported by a tool to i) identify which anomaly classes are most likely raised by the different attack families, ii) study suitability of anomaly detection algorithms to detect anomaly classes, iii) combine previous results to relate anomaly detection algorithms and attack families, and iv) define guidelines to select unsupervised algorithms for intrusion detection.

Service Oriented Architecture (SOA) is a popular design pattern that allows building applications composed of loosely-coupled and autonomous services. Such services may evolve and change at runtime, often outside the control of the owner of the application. Consequently, typical validation approaches, like offline testing performed before services deployment, are necessary but not sufficient: offline testing cannot assure the correct behavior of the SOA during its execution. To cope with the evolution of services and their orchestrations, in this paper we present a Self-Adaptive Model-BAsed online testing framework called SAMBA. SAMBA aims to assess the proper behavior of a SOA during its lifecycle executing model-based online testing at runtime, under the coordination of a MAPE-K control loop. SAMBA is assessed in a case study, where its detection capability are proved through functional, mutation and fault injection tests.

Anomaly detection aims at identifying patterns in data that do not conform to the expected behavior. Despite anomaly detection has been arising as one of the most powerful techniques to suspect attacks or failures, dedicated support for the experimental evaluation is actually scarce. In fact, existing frameworks are mostly intended for the broad purposes of data mining and machine learning. Intuitive tools tailored for evaluating anomaly detection algorithms for failure and attack detection with an intuitive support to sliding windows are currently missing. This paper presents RELOAD, a flexible and intuitive tool for the Rapid EvaLuation Of Anomaly Detection algorithms. RELOAD is able to automatically i) fetch data from an existing data set, ii) identify the most informative features of the data set, iii) run anomaly detection algorithms, including those based on sliding windows, iv) apply multiple strategies to features and decide on anomalies, and v) provide conclusive results following an extensive set of metrics, along with plots of algorithms scores. Finally, RELOAD includes a simple GUI to set up the experiments and examine results. After describing the structure of the tool and detailing inputs and outputs of RELOAD, we exercise RELOAD to analyze an intrusion detection dataset available on a public platform, showing its setup, metric scores and plots.

The growing systems complexity calls for dedicated monitoring and data analysis strategies aiming to detect faults, attacks and errors before they escalate into failures. Distributed and heterogeneous systems are more likely to expose vulnerabilities that attackers may target to get unauthorized access to a system, make it unavailable or steal sensitive data. As countermeasure, traditionally techniques for attacks and intrusion detection are based on signature recognition and requires knowledge on the attacks pattern: therefore, they are not well-suited to detect zero-days attacks. A viable alternative is anomaly detection, where deviation from the expected behavior are suspected as attacks. However, anomaly detection is generally not applicable in systems where the expected behavior changes through time. In this paper we explore anomaly detection strategies based on sliding windows, which are intended for evolving and dynamic systems as IoT, in which system configuration and behavior may change continuously. We first describe the context and the key features of sliding windows, and then we proceed detailing their possible drawbacks. Discussion is substantiated by quantitative analyses directed to evaluate detection capabilities. The experimental campaign is based on state-of-the-art algorithms and datasets, and results have been made publicly available.

Cyber-physical Systems of Systems (SoSs) are large-scale systems made of independent and autonomous cyber-physical Constituent Systems (CSs) which may interoperate to achieve high-level goals also with the intervention of humans. Providing security in such SoSs means, among other features, forecasting and anticipating evolving SoS functionalities, ultimately identifying possible detrimental phenomena that may result from the interactions of CSs and humans. Such phenomena, usually called emergent phenomena , are often complex and difficult to capture: the first appearance of an emergent phenomenon in a cyber-physical SoS is often a surprise to the observers. Adequate support to understand emergent phenomena will assist in reducing both the likelihood of design or operational flaws, and the time needed to analyze the relations amongst the CSs, which always has a key economic significance. This article presents a threat analysis methodology and a supporting tool aimed at (i) identifying (emerging) threats in evolving SoSs, (ii) reducing the cognitive load required to understand an SoS and the relations among CSs, and (iii) facilitating SoS risk management by proposing mitigation strategies for SoS administrators. The proposed methodology, as well as the tool, is empirically validated on Smart Grid case studies by submitting questionnaires to a user base composed of 3 stakeholders and 18 BSc and MSc students.

Anomaly detection, which aims at identifying unexpected trends and data patterns, has widely been used to build error detectors, failure predictors or intrusion detectors. Internal faults or malicious attacks have a different impact on the behavior of the system. They usually manifest as different observable deviations from the expected behavior, which may be identified by anomaly detection algorithms. Our study aims at investigating the suitability of unsupervised algorithms and their families in detecting either point, contextual or collective anomalies. To provide a complete picture, we consider both sliding and non-sliding window algorithms which operate in unsupervised mode. Along with qualitative analyses of each algorithm and family, we conduct an experimental campaign in which we run each algorithm on three state-of-the-art datasets in which we inject either point, contextual or collective anomalies. Results show that non-sliding algorithms are capable to detect point and collective anomalies, while they cannot effectively deal with contextual ones. Instead, sliding window algorithms require shorter periods of training and naturally build a local context, which allow them to effectively deal with contextual anomalies. Such observations are summarized to support the choice of the correct algorithm depending on the investigated class(es) of anomaly.

Abstract In recent years, more and more efforts have been devoted in supporting the design of systems‐of‐systems (SoS). Designing such systems is a multidisciplinary problem which involves considering emergent phenomena, assuring the achievement of dependability/security requirements, guaranteeing system responsiveness, and supporting dynamicity/evolution and multicriticality of provided services. A first step towards a viable design approach is to provide a conceptual model of SoS which captures SoS concepts, and their interrelationships aiming at enhancing the understandability of SoS to stakeholders and providing the basis for further automated analysis. In this context, the AMADEOS European project is bringing together researchers and practitioners to provide the support to design SoS starting from the definition of a domain specific ontology serving as a vocabulary for SoS. Our contribution consists in the modeling of the key SoS concepts and relationships defined in AMADEOS adopting a systems modeling language visual modeling language. We propose a systems modeling language profile for SoS, and we show its applicability in a Smart Grid scenario. We show how to use the profile in a model‐driven engineering process to support different types of analyses, and we discuss how to integrate the profile in a user‐friendly model‐driven engineering tool for SoS rapid modeling, validation, code‐generation, and simulation.

Abstract Thanks to the large availability of portable devices and the growing interest in the Internet of Things, during crises, social networks, or alerts sent through mobile devices or sensor networks are available and can be matched each other to perform situational analysis. However, the inclusion of multiple heterogeneous sources in situational analyses leads to 2 main issues: (1) a source could deliver (voluntarily or erroneously) wrong data damaging the integrity and the correctness of the analysis, and (2) a significant amount of heterogeneous data need to be processed. As a consequence, the crisis management operator faces a large amount of potentially unreliable data. In this paper, we present a relevance labelling strategy to process information gathered from heterogeneous data streams to select the most relevant events. These are presented to the crisis management operator with the highest priority. Our strategy is evaluated using events collected by the Secure! crisis management system, considering 3 real crisis scenarios happened in Italy in 2015. Results show that our strategy is able to correctly identify sets of relevant events, supporting the activities of the crisis management operator.

As our society massively relies on ICT, security services are becoming essential to protect users and entities involved. Amongst such services, non-repudiation provides evidences of actions, protects against their denial, and helps solving disputes between parties. For example, it prevents denial of past behaviors as having sent or received messages. Noteworthy, if the information flow is continuous, evidences should be produced for the entirety of the flow and not only at specific points. Further, non-repudiation should be guaranteed by mechanisms that do not reduce the usability of the system or application. To meet these challenges, in this paper, we propose two solutions for non-repudiation of remote services based on multi-biometric continuous authentication. We present an application scenario that discusses how users and service providers are protected with such solutions. We also discuss the technological readiness of biometrics for non-repudiation services: the outcome is that, under specific assumptions, it is actually ready.

Ensuring an external electricity supply to critical city components during flood events requires adequate urban grid planning. The proliferation of smart grid technologies means that such planning needs to assess how smart grids might function during floods. This paper proposes a method to qualitatively investigate robustness of electricity supply to smart grid consumers during flood events. This method builds on the Hazus methodology and aims to provide inputs for the risk analysis of urban grids.

User authentication is a key service, especially for systems that can be considered critical for the data stored and the functionalities offered. In those cases, traditional authentication mechanisms can be inadequate to face intrusions: they usually verify user's identity only at login, and even repeating this step, frequently asking for passwords or PIN would reduce system's usability. Biometric continuous authentication, instead, is emerging as viable alternative approach that can guarantee accurate and transparent verification for the entire session: the traits can be repeatedly acquired avoiding disturbing the user's activity. Another security service that these systems may need is nonrepudiation, which protect against the denial of having used the system or executed some commands with it. The paper focuses on biometric continuous authentication and nonrepudiation, and it briefly presents a preliminary solution based on a specific case study. This work presents the current research direction of the author and describes some challenges that the student aims to address in the next years.

Software infrastructures are becoming more and more complex, making performance and dependability monitoring in wide and dynamic contexts such as Distributed Systems, Systems of Systems (SoS) and Cloud environments an unachievable goal. Consequently, it is very difficult to know how all the specific parts, services and modules of these systems behave. This negatively impacts our ability in detecting anomalies, because the boundaries between normal and anomalous behaviors are not always known. The paper describes the context and the targeted problem highlighting the research directions that the student will follow in the next years. In particular, after introducing the relevance of this work with respect to the academic and the industrial state of the art, we carefully define the problem and summarize the main challenges that arise according to such problem definition.

The workshop on Recent Advances in the DependabIlity AssessmeNt of Complex systEms (RADIANCE), in its first edition, aims to discuss novel dependability assessment approaches for complex systems and to promote their adoption in real-world settings through industrial and academic research. The main objective is to promote and foster discussion on novel ideas, constituting a forum where researchers can share both real problems and innovative solutions for the assessment of complex systems. The workshop focuses on assessing complex evolving systems, where increasing complexity and changes are due to the introduction of new components and sensors, and to the extensive usage of software OTS components or black box components in general. In this macro area, the workshop welcomed a broad list of applications ranging from agile development in critical systems to model-driven assessment approaches as well as new needs for verification, validation and certification of dynamic and evolving systems, which also includes solutions for automating the verification and validation processes. Finally, the workshop was interested inexperimental assessment of dependability and security at large.

Session management in distributed Internet services is traditionally based on username and password, explicit logouts and mechanisms of user session expiration using classic timeouts. Emerging biometric solutions allow substituting username and password with biometric data during session establishment, but in such an approach still a single verification is deemed sufficient, and the identity of a user is considered immutable during the entire session. Additionally, the length of the session timeout may impact on the usability of the service and consequent client satisfaction. This paper explores promising alternatives offered by applying biometrics in the management of sessions. A secure protocol is defined for perpetual authentication through continuous user verification. The protocol determines adaptive timeouts based on the quality, frequency and type of biometric data transparently acquired from the user. The functional behavior of the protocol is illustrated through Matlab simulations, while model-based quantitative analysis is carried out to assess the ability of the protocol to contrast security attacks exercised by different kinds of attackers. Finally, the current prototype for PCs and Android smartphones is discussed.

Amongst the features of Service Oriented Architectures (SOAs), their flexibility, dynamicity, and scalability make them particularly attractive for adoption in the ICT infrastructure of organizations. Such features come at the cost of improved difficulty in monitoring the SOA for error detection: i) faults may manifest themselves differently due to services and SOA evolution, and ii) interactions between a service and its monitors may need reconfiguration at each service update. This calls for monitoring solutions that operate at different layers than the application layer (services layer). In this paper we present our ongoing work towards the definition of a monitoring framework for SOAs and services, which relies on anomaly detection performed at the Application Server (AS) and the Operating System (OS) layers to identify events whose manifestation or effect is not adequately described a-priori. Specifically the paper introduces the key concepts of our work and presents the case study built to exercise and set-up our monitor. The case study uses Life ray as application layer and it includes fault injection and data collection instruments to perform extended testing campaigns.

Highly distributed, autonomous and self-powered systems operating in harsh, outdoors environments face several threats in terms of dependability, timeliness and security, due to the challenging operating conditions determined by the environment. Despite such difficulties, there is an increasing demand to deploy these systems to support critical services, thus calling for severe timeliness, safety, and security requirements. Several challenges need to be faced and overcome. First, the designed architecture must be able to cope with the environmental challenges and satisfy dependability, timeliness and security requirements. Second, the assessment of the system must be carried on despite potentially incomplete field-data, and complex cascading effects that small modifications in system properties and operating conditions may have on the targeted metrics. In this paper we present our experience from the EU-funded project ALARP (A railway automatic track warning system based on distributed personal mobile terminals), which aims to build and validate a distributed, real-time, safety-critical system that detects trains approaching a railway worksite and notifies their arrivals to railway trackside workers. The paper describes the challenges we faced, and the solutions we adopted, when architecting and evaluating the ALARP system.

Session management in distributed Internet services is traditionally based on username and password, and explicit logouts and timeouts that expire due to idle activity of the user. Emerging biometric solutions allow substituting username and password with biometric data, but still a single verification is deemed sufficient, and the identity of a user is considered immutable during the entire session. Additionally, the length of the timeout may impact on the usability of the service and consequent client satisfaction. This paper explores promising alternatives offered by biometrics for the management of sessions. A secure protocol is defined for perpetual authentication through continuous user verification. The protocol determines adaptive timeouts selected on the basis of the quality, frequency and type of biometric data acquired transparently from the user. Protocol behavior is shown through simulations.

Performing decentralized network fault diagnosis based on network traffic is challenging. Besides inherent stochastic behaviour of observations, measurements may be subject to errors degrading diagnosis timeliness and accuracy. In this paper we present a novel approach in which we aim to mitigate issues of measurement errors by quantifying uncertainty. The uncertainty information is applied in the diagnostic component to improve its robustness. Three diagnosis components have been proposed based on the Hidden Markov Model formalism: (H0) representing a classical approach, (H1) a static compensation of (H0) to uncertainties and (H2) dynamically adapting diagnosis to uncertainty information. From uncertainty injection scenarios of added measurement noise we demonstrate how using uncertainty information can provide a structured approach of improving diagnosis.

A software clock capable of self-evaluating its synchronization uncertainty is experimentally validated for a specific implementation on a node synchronized through NTP. The validation methodology takes advantage of an external node equipped with a GPS-synchronized clock acting as a reference, which is connected to the node hosting the system under test through a fast Ethernet connection. Experiments are carried out for different values of the software clock parameters and different types of workload, and address the possible occurrence of faults in the system under test and in the NTP synchronization mechanism. The validation methodology is designed to be as less intrusive as possible and to grant a resolution of the order of few hundreds of microseconds. The experimental results show very good performance of R&SAClock, and their analysis gives precious hints for further improvements.

Experimental evaluation is aimed at providing useful insights and results that constitute a confident representation of the system under evaluation. Although guidelines and good practices exist and are often applied, the uncertainty of results and the quality of the measuring system is rarely discussed. To complement such guidelines and good practices in experimental evaluation, metrology principles can contribute in improving experimental evaluation activities by assessing the measuring systems and the results achieved. In this paper we present the experimental evaluation by software-implemented fault injection of a safe train-borne driver machine interface (DMI), to evaluate its behavior in presence of faults. The measuring system built for the purpose and the results obtained on the assessment of the DMI are scrutinized along basic principles of metrology and good practices of fault injection. Trustfulness in results has been estimated satisfactory and the experimental campaign has shown that the safety mechanisms of the DMI correctly identify the faults injected and that a proper reaction is executed.

In many distributed and pervasive systems the clocks of nodes are required to be synchronized to a unique global time. Due to unpredictable system and environment characteristics, the distance of a local clock from global time is a variable factor very hard to predict. Systems usually adopt measures to guarantee an upper bound on such distance from global time that are very often quite far from typical execution scenarios and thus are of practical little use. As a consequence, while in many circumstances reliable information on the actual distance from global time would improve system behaviour, unfortunately such information is usually not available. In this paper we propose the Reliable and Self-Aware Clock (R&SAClock), a low-intrusive software service that is able to compute a conservative estimation of distance from an external global time. R&SAClock acts as a new clock that couples information gained from synchronization mechanisms with information collected from the local clock to provide both current time and a self-adaptive reliable estimation of distance from global time. This paper describes the R&SAClock as a system component: we define its main functions, services and time-related mechanisms. Finally details of an implementation of the R&SAClock for the NTP synchronization mechanism and Linux OS are shown.

Increasing interest is being paid to quantitative evaluation based on measurements of dependability attributes and metrics of computer systems and infrastructures. Despite measurands are generally sensibly identified, different approaches make it difficult to compare different results. Moreover, measurement tools are seldom recognized for what they are: measuring instruments. In this paper, many measurement tools, present in the literature, are critically evaluated at the light of metrology concepts and rules. With no claim of being exhaustive, the paper (i) investigates if and how deeply such tools have been validated in accordance to measurement theory, and (ii) tries to evaluate (if possible) their measurement properties. The intention is to take advantage of knowledge available in a recognized discipline such as metrology and to propose criteria and indicators taken from such discipline to improve the quality of measurements performed in evaluation of dependability attributes.

NekoStat is a Java framework and tool developed for qualitative and quantitative evaluation of dependability attributes of distributed algorithms. In this paper, NekoStat is analyzed along the lines of metrology. First the relevant metrological properties that a tool such as NekoStat should possess are introduced. The lack of a rigorous metrological characterization of the accuracy of collected measures is noticed as there is no estimation of how biased the collected data can be. To solve this, a new component, called OffsetDetector, is introduced and described. OffsetDetector allows to estimate the uncertainty of collected data and enables NekoStat to be aware of the accuracy level of the local clocks during distributed executions. The collected time measurements can thus be distinguished depending on the synchronization quality at the instant they were collected. In this way, the trustworthiness in the results is widely enhanced as shown through a case study illustrated in the paper

The paper addresses the challenges and opportunities of instrumenting pervasive computing systems with a logical clock, aware of the quality of synchronization with respect to a time reference. Pervasive computing systems are: i) mobile; ii) dynamic; and iii) composed of a large number of distributed components; in systems with these characteristics the availability of a "smart" clock that is: i) capable to use different mechanisms for the synchronization with the global distributed time reference; and ii) aware of the current quality of synchronization with such time reference, can be very useful in order to build dependable middleware services and applications