Medical cyber-physical systems (MCPS) are increasingly adopting learning-enabled components (LECs) to enhance their decision-making capabilities. Due to the safety-critical nature of MCPS, these systems must maintain high performance on both expected and unexpected input data. Therefore, ensuring the robustness of LE-MCPS is crucial for their successful deployment. Existing research predominantly focuses on robustness to synthetic adversarial examples , crafted by adding imperceptible perturbations to clean input data. However, these synthetic adversarial examples do not accurately reflect the most challenging real-world scenarios, especially in the context of healthcare data. Consequently, robustness to synthetic adversarial examples may not necessarily translate to robustness against naturally occurring adversarial examples . We propose a method to evaluate the robustness of LE-MCPS to natural adversarial examples. The method curates naturally adversarial datasets leveraging probabilistic labels obtained from automated weakly supervised labeling which combines noisy and cheap-to-obtain labeling heuristics. Based on these labels, the method adversarially orders the input data and uses this ordering to construct a sequence of increasingly adversarial datasets for assessing robustness. Our evaluation on six MCPS case studies and two non-medical case studies demonstrates (1) the efficacy and statistical validity of our approach to generating naturally adversarial datasets and (2) the utility of our robustness evaluation in classifying robust and non-robust LE-MCPS.

Advances in wearable technology have enabled ubiquitous use of wearable devices in remote patient monitoring, particularly in clinical trials. Because of the reliance on highquality data in these endeavors, the first and often the most time-consuming step is to build a data collection system. While many systems have been developed to address this, they are often highly specific and customized to the task at hand, and are often not generalized enough to support other tasks. To remedy this, we developed Raproto, an open-source easy-to-use rapid prototyping platform that does not require the time, effort, and expertise needed for custom development. The Raproto platform consists of three components, the wearable device(s), communication protocol, and remote storage. These components support the collection, transmission, storage, analysis, and visualization of large-scale data with applications from smaller-scale research studies to large clinical trials. To reduce the burden of device and application development, we created multipurpose and customizable smartwatch applications on both the Android and Tizen operating systems. We evaluate our platform in a lab setting as well as in two real-world case studies. Overall, we find that we can collect data using our application for over 24 hours on a single charge and there is little to no data loss, thus making it an ideal tool to preface customized device development for real-world impact and commercialization.

Imitation learning considerably simplifies policy synthesis compared to alternative approaches by exploiting access to expert demonstrations. For such imitation policies, errors away from the training samples are particularly critical. Even rare slip-ups in the policy action outputs can compound quickly over time, since they lead to unfamiliar future states where the policy is still more likely to err, eventually causing task failures. We revisit simple supervised "behavior cloning" for conveniently training the policy from nothing more than pre-recorded demonstrations, but carefully design the model class to counter the compounding error phenomenon. Our "memory-consistent neural network" (MCNN) outputs are hard-constrained to stay within clearly specified permissible regions anchored to prototypical "memory" training samples. We provide a guaranteed upper bound for the sub-optimality gap induced by MCNN policies. Using MCNNs on 10 imitation learning tasks, with MLP, Transformer, and Diffusion backbones, spanning dexterous robotic manipulation and driving, proprioceptive inputs and visual inputs, and varying sizes and types of demonstration data, we find large and consistent gains in performance, validating that MCNNs are better-suited than vanilla deep neural networks for imitation learning applications. Website: https://sites.google.com/view/mcnn-imitation

Social interaction quality ratings derived from short natural conversations can differentiate children with and without autism at the group level. In this work, we explored conversations between children and an unfamiliar adult who rated their social interaction success on six dimensions. Using hand-crafted acoustic and lexical features, we built different classifiers to predict children's dimensional conversation quality. The best classifier achieved 61% accuracy, which outperformed human raters (49%). Follow-up analyses revealed that a subset of features determined communication quality scores. Additionally, we extracted acoustic features using a pretrained audio transformer and improved our prediction to 68%. This study suggests that automatically predicting conversation quality could be an inexpensive and objective way to monitor intervention progress in children with communication challenges, and could be used to identify intervention targets for improving conversational success.

Classification of clinical alarms is at the heart of prioritization, suppression, integration, postponement, and other methods of mitigating alarm fatigue. Since these methods directly affect clinical care, alarm classifiers, such as intelligent suppression systems, need to be evaluated in terms of their sensitivity and specificity, which is typically calculated on a labeled dataset of alarms. Unfortunately, the collection and particularly labeling of such datasets requires substantial effort and time, thus deterring hospitals from investigating mitigations of alarm fatigue. This article develops a lightweight method for evaluating alarm classifiers without perfect alarm labels. The method relies on probabilistic labels obtained from data programming—a labeling paradigm based on combining noisy and cheap-to-obtain labeling heuristics. Based on these labels, the method produces confidence bounds for the sensitivity/specificity values from a hypothetical evaluation with manual labeling. Our experiments on five alarm datasets collected at Children’s Hospital of Philadelphia show that the proposed method provides accurate bounds on the classifier’s sensitivity/specificity, appropriately reflecting the uncertainty from noisy labeling and limited sample sizes.

Providing reliable model uncertainty estimates is imperative to enabling robust decision making by autonomous agents and humans alike. While recently there have been significant advances in confidence calibration for trained models, examples with poor calibration persist in most calibrated models. Consequently, multiple techniques have been proposed that leverage label-invariant transformations of the input (i.e., an input manifold) to improve worst-case confidence calibration. However, manifold-based confidence calibration techniques generally do not scale and/or require expensive retraining when applied to models with large input spaces (e.g., ImageNet). In this paper, we present the recursive lossy label-invariant calibration (ReCal) technique that leverages label-invariant transformations of the input that induce a loss of discriminatory information to recursively group (and calibrate) inputs – without requiring model retraining. We show that ReCal outperforms other calibration methods on multiple datasets, especially, on large-scale datasets such as ImageNet.

Abstract This paper presents Verisig 2.0, a verification tool for closed-loop systems with neural network (NN) controllers. We focus on NNs with tanh/sigmoid activations and develop a Taylor-model-based reachability algorithm through Taylor model preconditioning and shrink wrapping. Furthermore, we provide a parallelized implementation that allows Verisig 2.0 to efficiently handle larger NNs than existing tools can. We provide an extensive evaluation over 10 benchmarks and compare Verisig 2.0 against three state-of-the-art verification tools. We show that Verisig 2.0 is both more accurate and faster, achieving speed-ups of up to 21x and 268x against different tools, respectively.

High-quality labels are expensive to obtain for many machine learning tasks, such as medical image classification tasks. Therefore, probabilistic (weak) labels produced by weak supervision tools are used to seed a process in which influential samples with weak labels are identified and cleaned by several human annotators to improve the model performance. To lower the overall cost and computational overhead of this process, we propose a solution called CHEF (CHEap and Fast label cleaning), which consists of the following three components. First, to reduce the cost of human annotators, we use INFL, which prioritizes the most influential training samples for cleaning and provides cleaned labels to save the cost of one human annotator. Second, to accelerate the sample selector phase and the model constructor phase, we use Increm-INFL to incrementally produce influential samples, and DeltaGrad-L to incrementally update the model. Third, we redesign the typical label cleaning pipeline so that human annotators iteratively clean smaller batch of samples rather than one big batch of samples. This yields better overall model performance and enables possible early termination when the expected model performance has been achieved. Extensive experiments show that our approach gives good model prediction performance while achieving significant speed-ups.

This article addresses the problem of verifying the safety of autonomous systems with neural network (NN) controllers. We focus on NNs with sigmoid/tanh activations and use the fact that the sigmoid/tanh is the solution to a quadratic differential equation. This allows us to convert the NN into an equivalent hybrid system and cast the problem as a hybrid system verification problem, which can be solved by existing tools. Furthermore, we improve the scalability of the proposed method by approximating the sigmoid with a Taylor series with worst-case error bounds. Finally, we provide an evaluation over four benchmarks, including comparisons with alternative approaches based on mixed integer linear programming as well as on star sets.

Reliable uncertainty estimates are an important tool for helping autonomous agents or human decision makers understand and lever-age predictive models. However, existing approaches to estimating uncertainty largely ignore the possibility of covariate shift—i.e.,where the real-world data distribution may differ from the training distribution. As a consequence, existing algorithms can overestimate certainty, possibly yielding a false sense of confidence in the predictive model. We pro-pose an algorithm for calibrating predictions that accounts for the possibility of covariate shift, given labeled examples from the train-ing distribution and unlabeled examples from the real-world distribution. Our algorithm uses importance weighting to correct for the shift from the training to the real-world distribution. However, importance weighting relies on the training and real-world distributions to be sufficiently close. Building on ideas from domain adaptation, we additionally learn a feature map that tries to equalize these two distributions. In an empirical evaluation, we show that our proposed approach outperforms existing approaches to calibrated prediction when there is covariate shift.

This paper describes a verification case study on an autonomous racing car with a neural network (NN) controller. Although several verification approaches have been recently proposed, they have only been evaluated on low-dimensional systems or systems with constrained environments. To explore the limits of existing approaches, we present a challenging benchmark in which the NN takes raw LiDAR measurements as input and outputs steering for the car. We train a dozen NNs using reinforcement learning (RL) and show that the state of the art in verification can handle systems with around 40 LiDAR rays. Furthermore, we perform real experiments to investigate the benefits and limitations of verification with respect to the sim2real gap, i.e., the difference between a system's modeled and real performance. We identify cases, similar to the modeled environment, in which verification is strongly correlated with safe behavior. Finally, we illustrate LiDAR fault patterns that can be used to develop robust and safe RL algorithms.

Model-based design offers a promising approach for assisting developers to build reliable and secure cyber-physical systems in a systematic manner. In this methodology, a designer first constructs a model, with mathematically precise semantics, of the system under design, and performs extensive analysis with respect to correctness requirements before generating the implementation from the model. However, as new vulnerabilities are discovered, requirements evolve aimed at ensuring resiliency. There is currently a shortage of an inexpensive, automated software that can effectively repair the initial design, and a model-based system developer regularly needs to redesign and reimplement the system from scratch. In this paper, we propose a new methodology along with a MATLAB software called REAFFIRM to facilitate the model-based repair for improving the resiliency of cyber-physical systems. REAFFIRM takes as inputs 1) an original hybrid system modeled as a Simulink/Stateflow diagram, 2) a given resiliency pattern specified as a model transformation script, and 3) a safety requirement expressed as a Signal Temporal Logic formula, and outputs a repaired model which satisfies the requirement. The tool consists of two main modules, model transformation followed by model synthesis. While the latter component is built on top of the falsification tool Breach, to implement the former, we introduce a new model transformation language for hybrid systems, which we call HATL, to allow a designer to specify resiliency patterns. To evaluate the proposed approach, we use REAFFIRM to automatically synthesize the repaired models of four different case studies.

Runtime monitoring is a vital part of safety-critical systems. However, early stage assurance of monitoring quality is currently limited: it relies either on complex models that might be inaccurate in unknown ways or on data that would only be available once the system has been built. To address this issue, we propose a compositional framework for modeling and analysis of noisy monitoring systems. Our novel 3-value detector model uses probability spaces to represent atomic (noncomposite) detectors, and it composes them into a temporal logic-based monitor. The error rates of these monitors are estimated by our analysis engine, which combines symbolic probability algebra, independence inference, and estimation from labeled detection data. Our evaluation on an autonomous underwater vehicle found that our framework produces accurate estimates of error rates while using only detector traces, without any monitor traces. Furthermore, when data are scarce, our approach shows higher accuracy than noncompositional data-driven estimates from monitor traces. Thus, this article enables accurate evaluation of logical monitors in early design stages before deploying them.

This paper presents Verisig, a hybrid system approach to verifying safety properties of closed-loop systems using neural networks as controllers. We focus on sigmoid-based networks and exploit the fact that the sigmoid is the solution to a quadratic differential equation, which allows us to transform the neural network into an equivalent hybrid system. By composing the network's hybrid system with the plant's, we transform the problem into a hybrid system verification problem which can be solved using state-of-the-art reachability tools. We show that reachability is decidable for networks with one hidden layer and decidable for general networks if Schanuel's conjecture is true. We evaluate the applicability and scalability of Verisig in two case studies, one from reinforcement learning and one in which the neural network is used to approximate a model predictive controller.

Information flow analysis is an effective way to check useful security properties, such as whether secret information can leak to adversaries. Despite being widely investigated in the realm of programming languages, information-flow-based security analysis has not been widely studied in the domain of cyber-physical systems (CPS). CPS provide interesting challenges to traditional type-based techniques, as they model mixed discrete-continuous behaviors and are usually expressed as a composition of state machines. In this paper, we propose a lightweight static analysis methodology that enables information security properties for CPS models. We introduce a set of security rules for hybrid automata that characterizes the property of non-interference. Based on those rules, we propose an algorithm that generates security constraints between each sub-component of hybrid automata, and then transforms these constraints into a directed dependency graph to search for non-interference violations. The proposed algorithm can be applied directly to parallel compositions of automata without resorting to model-flattening techniques. Our static checker works on hybrid systems modeled in Simulink/Stateflow format and decides whether or not the model satisfies non-interference given a user-provided security annotation for each variable. Moreover, our approach can also infer the security labels of variables, allowing a designer to verify the correctness of partial security annotations. We demonstrate the potential benefits of the proposed methodology on two case studies.

The disruptive vision of Medical Cyber-Physical Systems (MCPS) enables the promising next-generation of eHealth systems that are intended to interoperate efficiently, safely, and securely. Safety-critical interconnected systems that analyze patients' vital signs gathered from medical devices, infer the state of the patient's health, and start treatments issuing information to doctors and medical actuators should improve the patients' safety in a cost-efficient fashion. Despite the benefits provided by the MCPS vision, it also opens the door to critical challenges like the security and privacy, Quality of Service (QoS), and high availability of the devices composed to support the MCPS scenario. The Integrated Clinical Environment (ICE) standard is a significant step toward promoting open coordination of heterogeneous medical devices by considering the previous challenges. However, a lot of effort is still required in order to cover the whole aspects of these challenges and enable the future eHealth. In this context, we identify critical shortcomings of ICE using challenge scenarios regarding security, QoS, and high availability. According to these concerns and following the ICE standard, we propose the novel ICE++ architecture, which is oriented to the Mobile Edge Computing paradigm and combines SDN and NFV techniques to manage efficiently and automatically the MCPS elements taking into account its security, QoS, and high availability. Finally, we perform experiments that demonstrate the potential usefulness of our solution regarding the efficient and automatic management of the ICE components.

Signal Temporal Logic (STL) is a prominent specification formalism for real-time systems, and monitoring these specifications, specially when (for different reasons such as learning) behavior of systems can change over time, is quite important. There are three main challenges in this area: (1) full observation of system state is not possible due to noise or nuisance parameters, (2) the whole execution is not available during the monitoring, and (3) computational complexity of monitoring continuous time signals is very high. Although, each of these challenges has been addressed by different works, to the best of our knowledge, no one has addressed them all together. In this paper, we show how to extend any parameter invariant test procedure for single points in time to a parameter invariant test procedure for efficiently monitoring continuous time executions of a system against STL properties. We also show, how to extend probabilistic error guarantee of the input test procedure to a probabilistic error guarantee for the constructed test procedure.

The Internet of Medical Things (IoMT) is poised to revolutionize medicine. However, medical device communication, coordination, and interoperability present challenges for IoMT applications due to safety, security, and privacy concerns. These challenges can be addressed by developing an open platform for IoMT that can provide guarantees on safety, security and privacy. As a first step, we introduce OpenICE-lite, a middleware for medical device interoperability that also provides security guarantees and allows other IoMT applications to view/analyze the data in real time. We describe two applications that currently utilize OpenICE-lite, namely (i) a critical pulmonary shunt predictor for infants during surgery; (ii) a remote pulmonary monitoring systems (RePulmo). Implementations of both systems are utilized by the Children's Hospital of Philadelphia (CHOP) as quality improvements to patient care.

The tight interaction between information technology and the physical world inherent in cyber-physical systems (CPS) can challenge traditional approaches for monitoring safety and security. Data collected for robust CPS monitoring is often sparse and may lack rich training data describing critical events/attacks. Moreover, CPS often operate in diverse environments that can have significant inter/intra-system variability. Furthermore, CPS monitors that are not robust to data sparsity and inter/intra-system variability may result in inconsistent performance and may not be trusted for monitoring safety and security. Towards overcoming these challenges, this paper presents recent work on the design of parameter-invariant (PAIN) monitors for CPS. PAIN monitors are designed such that unknown events and system variability minimally affect the monitor performance. This work describes how PAIN designs can achieve a constant false alarm rate (CFAR) in the presence of data sparsity and intra/inter system variance in real-world CPS. To demonstrate the design of PAIN monitors for safety monitoring in CPS with different types of dynamics, we consider systems with networked dynamics, linear-time invariant dynamics, and hybrid dynamics that are discussed through case studies for building actuator fault detection, meal detection in type I diabetes, and detecting hypoxia caused by pulmonary shunts in infants. In all applications, the PAIN monitor is shown to have (significantly) less variance in monitoring performance and (often) outperforms other competing approaches in the literature. Finally, an initial application of PAIN monitoring for CPS security is presented along with challenges and research directions for future security monitoring deployments.

This article is concerned with the security of modern Cyber-Physical Systems in the presence of transient sensor faults. We consider a system with multiple sensors measuring the same physical variable, where each sensor provides an interval with all possible values of the true state. We note that some sensors might output faulty readings and others may be controlled by a malicious attacker. Differing from previous works, in this article, we aim to distinguish between faults and attacks and develop an attack detection algorithm for the latter only. To do this, we note that there are two kinds of faults—transient and permanent; the former are benign and short-lived, whereas the latter may have dangerous consequences on system performance. We argue that sensors have an underlying transient fault model that quantifies the amount of time in which transient faults can occur. In addition, we provide a framework for developing such a model if it is not provided by manufacturers. Attacks can manifest as either transient or permanent faults depending on the attacker’s goal. We provide different techniques for handling each kind. For the former, we analyze the worst-case performance of sensor fusion over time given each sensor’s transient fault model and develop a filtered fusion interval that is guaranteed to contain the true value and is bounded in size. To deal with attacks that do not comply with sensors’ transient fault models, we propose a sound attack detection algorithm based on pairwise inconsistencies between sensor measurements. Finally, we provide a real-data case study on an unmanned ground vehicle to evaluate the various aspects of this article.

Since Cyber Physical Systems (CPS) are widely used inmany safety-critical domains these days, critical properties such as robustness and resilience are required for such systems. To increase the robustness and the resilience of CPS, various sensor fusion techniques have been studied [1], [2],[3], [4]. These fusion techniques are based on certain sensor models, which broadly fall into two categories: probabilistic model and abstract model. The probabilistic sensor model [1] uses certain noise distributions on sensors (e.g., Gaussian), which is wellsuited for analyzing the systems' expected performance inthe average case. However, wrong assumptions on noise distributions may be in danger of being vulnerable to sensor attacks. On the other hand, the abstract sensor model [2] usesthe worst-case error bound of sensors. Thus, this model iswell suited for the systems' worst-case performance, whichis highly relevant to the case of sensor attacks [3].In this work, we study a hybrid sensor fusion that usesboth probabilistic and abstract sensor models to be able tobenefit from both. We demonstrate the validation of ourhybrid sensor fusion technique using an unmanned groundvehicle called Jackal.

In this paper we present a methodology to control ground robots under malicious attack on sensors. Within the term attack we intend any malicious disturbance injection on sensors, actuators, and controller that would compromise the safety of a robot. In order to guarantee resilience against attacks, we use a control-level technique implemented within a recursive algorithm that takes advantage of redundancy in the information received by the controller. We use the case study of a vehicle cruise-control, however, the strategy we present in this work is general for several applications. Our methodology relays on redundancy in the sensor measurements: specifically we consider N velocity measurements and use a recursive filtering technique that estimates the state of the system while being resilient against sensor attacks by acting on the variance of the measurements noise. Finally, we move our focus on hardware validation demonstrating our algorithm through extensive outdoor experiments conducted on two unmanned ground robots.