Robots operating in unstructured human environments inevitably encounter failures, especially in robot caregiving scenarios. While humans can often help robots recover, excessive or poorly targeted queries impose unnecessary cognitive and physical workload on the human partner. We present a human-in-the-loop failure-recovery framework for modular robotic policies, where a policy is composed of distinct modules such as perception, planning, and control, any of which may fail and often require different forms of human feedback. Our framework integrates calibrated estimates of module-level uncertainty with models of human intervention cost to decide which module to query and when to query the human. It separates these two decisions: a module selector identifies the module most likely responsible for failure, and a querying algorithm determines whether to solicit human input or act autonomously. We evaluate several module-selection strategies and querying algorithms in controlled synthetic experiments, revealing trade-offs between recovery efficiency, robustness to system and user variables, and user workload. Finally, we deploy the framework on a robot-assisted bite acquisition system and demonstrate, in studies involving individuals with both emulated and real mobility limitations, that it improves recovery success while reducing the workload imposed on users. Our results highlight how explicitly reasoning about both robot uncertainty and human effort can enable more efficient and user-centered failure recovery in collaborative robots. Supplementary materials and videos can be found at: emprise.cs.cornell.edu/modularhil.

Robot-assisted bite acquisition involves picking up food items with varying shapes, compliance, sizes, and textures. Fully autonomous strategies may not generalize efficiently across this diversity. We propose leveraging feedback from the care recipient when encountering novel food items. However, frequent queries impose a workload on the user. We formulate human-in-the-loop bite acquisition within a contextual bandit framework and introduce LINUCB-QG, a method that selectively asks for help using a predictive model of querying workload based on query types and timings. This model is trained on data collected in an online study involving 14 participants with mobility limitations, 3 occupational therapists simulating physical limitations, and 89 participants without limitations. We demonstrate that our method better balances task performance and querying workload compared to autonomous and always-querying baselines and adjusts its querying behavior to account for higher workload in users with mobility limitations. We validate this through experiments in a simulated food dataset and a user study with 19 participants, including one with severe mobility limitations. Please check out our project website at: emprise.cs.cornell.edu/hilbiteacquisition/.

Networks-on-chip (NoCs) form the backbone fabric for connecting multi-core SoCs containing several processor cores and memories. Design-space exploration (DSE) of NoCs is a crucial part of the SoC design process to ensure that it does not become a bottleneck. DSE today is often hindered by the inherent trade-off between software simulation vs hardware emulation/e- valuation. Software simulators are easily extendable and allow for the evaluation of new ideas but are not able to capture the hardware complexity. Meanwhile, RTL development is known to be time-consuming. This has forced DSE to use simulators followed by RTL development, evaluation and feedback, which slows down the overall design process. In an effort to tackle this problem, we present Proteus, a configurable and modular NoC simulator and RTL generator. Proteus is the first of its kind framework to use HLS compiler to develop NoCs from a C++ description of the N oC circuit. These generated N oCs can be simulated in software and tested on FPGAs. This allows users to do rapid DSE by providing the opportunity to tweak and test NoC architectures in real-time. We also compare Proteus-generated RTL with Chisel- generated and hand-written RTL in terms of area, timing and productivity. The ability to synthesize the NoC design on FPGAs can benefit large designs as the custom hardware results in faster run-time than cycle-accurate software simulators. Proteus is modeled similar to existing state-of-the-art simulators and offers users modifiable parameters to generate custom topologies, routing algorithms, and router microarchitectures.

This paper presents a novel approach of generating synthetic Photoplethysmogram (PPG) data using a physical model of the cardiovascular system to improve classifier performance with a combination of synthetic and real data. The physical model is an in-silico cardiac computational model, consisting of a four-chambered heart with electrophysiology, hemodynamic, and blood pressure auto-regulation functionality. Starting with a small number of measured PPG data, the cardiac model is used to synthesize healthy as well as PPG time-series pertaining to coronary artery disease (CAD) by varying pathophysiological parameters. A Variational Autoencoder (VAE) structure is proposed to derive a statistical feature space for CAD classification. Results are presented in two perspectives namely, (i) using artificially reduced real disease data and (ii) using all the real disease data. In both cases, by augmenting with the synthetic data for training, the performance (sensitivity, specificity) of the classifier changes from (i) (0.65, 1) to (1, 0.9) and (ii) (1, 0.95) to (1, 1). The proposed hybrid approach of combining physical modelling and statistical feature space selection generates realistic PPG data with pathophysiological interpretation and can outperform a baseline Generative Adversarial Network (GAN) architecture with a relatively small amount of real data for training. This proposed method could aid as a substitution technique for handling the problem of bulk data required for training machine learning algorithms for cardiac health-care applications.

Abnormal heart sounds may have diverse frequency characteristics depending upon underlying pathological conditions. Designing a binary classifier for predicting normal and abnormal heart sounds using supervised learning requires a lot of training data, covering different types of cardiac abnormalities. In this paper, we propose a semi-supervised approach to solve the problem. A convolutional Variational Autoencoder (VAE) structure is defined for learning the probability distribution of the spectrogram properties of normal heart sounds. The Kullback-Leibler (KL) divergence between the known prior distribution of the VAE and the encoded distribution is taken as an anomaly score for detecting abnormal heart sounds. The proposed approach is evaluated on open access and in-house datasets of Phonocardiogram (PCG) signals, recorded from normal subjects and patients, having cardiovascular diseases, cardiac murmurs and extra heart sounds. Results show that an improved classification performance is achieved in comparison to the existing approaches.

Coronary Artery Disease (CAD) causes significant global mortality. The recent development in artificial intelligence shows the feasibility of early non-invasive screening of several life- threatening cardiovascular diseases. However, such approaches have been less prolific in diagnosis of CAD due to lack of clinically known definite bio-marker. In this paper, we propose a novel neural network architecture that effectively combines two non-specific CAD markers, 1) anomalous morphology of Electrocardiogram (ECG) waveform and 2) abnormal Heart Rate Variability (HRV). A Convolutional Neural Network (CNN) structure is defined for extraction of morphological ECG features. Another composite structure is defined based on Long Short-Term Memory (LSTM) and a set of hand crafted statistical features for measuring the extent of HRV. The two independent bio-markers are subsequently combined in a hybrid CNN-LSTM architecture for classification of CAD. The proposed approach is evaluated on two datasets, a corpus, selected from the MIMIC II waveform dataset and a partially noisy in-house dataset, recorded using a low-cost ECG sensor. Results show that overall classification accuracy of 93% and 88% are achieved on the two datasets, which outperform the existing approaches.

In this paper we propose a novel process flow of a low-cost, non-invasive screening system for identifying Coronary Artery Disease (CAD) patients using a two-stage classification approach. A statistical rule engine is designed based on patient demography and medical history which is applied at the first stage of the proposed classification system. The misclassification error at this stage is reduced at the second stage based on numerical features extracted from multiple cardiovascular signals. Two sets of features are extracted from phonocardiogram (PCG) and photoplethysmogram (PPG) signals, collected from each subject for creating two independent Support Vector Machine (SVM) classifiers. Outcomes of the two classifiers are fused at the decision level for final prediction at second stage based on absolute distance of the test data-point from its respective SVM hyperplane. Results show that the proposed approach achieves sensitivity of 0.92 and specificity of 0.90 in classifying CAD patients on a hospital dataset of 99 subjects including CAD and non-CAD patients.

In this paper we propose a feature extraction algorithm for classifying Coronary Artery Disease (CAD) patients from Photoplethysmogram (PPG) signals. Several domain-independent features, representing inherent properties of a time series are explored in our study. These are combined with Heart Rate Variability (HRV) and other popularly used morphological PPG features. A statistical feature selection algorithm, based on Maximal Information Coefficient (MIC) is applied on MIMIC II dataset for ranking and choosing the optimum features. A second hospital dataset of different patient demography is used for performance evaluation. Results show that, Support Vector Machine (SVM) classifier, designed using the selected features yields average sensitivity and specificity of more than 0.8 in identifying CAD patients and also outperforms two recent prior art approaches when applied on the test dataset.

This paper presents a demo proposal of a standalone smartphone application that can automatically analyse the signal quality of PCG, as it is recorded on a low-cost smartphone-based digital stethoscope. Features, related to the inherent pattern of the autocorrelated signal envelope, have been used for classifying and discarding the noisy portions from a continuous PCG. Our application has been successfully deployed on Nexus 5 and tested on several clean and noisy PCG signals with sensitivity 78.91% and specificity 70.83%.

Non-invasive cuff-less Blood Pressure (BP) estimation from Photoplethysmogram (PPG) is a well known challenge in the field of affordable healthcare. This paper presents a set of improvements over an existing method that estimates BP using 2-element Windkessel model from PPG signal. A noisy PPG corpus is collected using fingertip pulse oximeter, from two different locations in India. Exhaustive pre-processing techniques, such as filtering, baseline and topline correction are performed on the noisy PPG signals, followed by the selection of consistent cycles. Subsequently, the most relevant PPG features and demographic features are selected through Maximal Information Coefficient (MIC) score for learning the latent parameters controlling BP. Experimental results reveal that overall error in estimating BP lies within 10% of a commercially available digital BP monitoring device. Also, use of alternative latent parameters that incorporate the variation in cardiac output, shows a better trend following for abnormally low and high BP.

Heart rate variability (HRV) measures the instantaneous change in heart rate and is an important marker for checking physical condition as well as mental stress of a person. In this paper, we propose a methodology to calculate HRV of a person using smart phone audio. Heart sound is captured in the inbuilt microphone of a smart phone, by placing the device on the chest of the person. We propose a process flow to make the phone captured noisy audio signal clean and audible. Furthermore, we propose a novel peak detection algorithm for accurately locating the peaks corresponding to heart sound in the noisy audio signal. The algorithm is also capable of rejecting the noisy peaks present in the captured audio that resembles heart sound pattern. Results show that the proposed methodology yields significant improvement in estimating HRV parameters compared to a clinical pulse-oximeter device, that works on the principle of photoplethysmogram (PPG) technique.

Photoplethysmography (PPG) signals, captured using smart phones are generally noisy in nature. Although they have been successfully used to determine heart rate from frequency domain analysis, further indirect markers like blood pressure (BP) require time domain analysis for which the signal needs to be substantially cleaned. In this paper we propose a methodology to clean such noisy PPG signals. Apart from filtering, the proposed approach reduces the baseline drift of PPG signal to near zero. Furthermore it models each cycle of PPG signal as a sum of 2 Gaussian functions which is a novel contribution of the method. We show that, the noise cleaning effect produces better accuracy and consistency in estimating BP, compared to the state of the art method that uses the 2-element Windkessel model on features derived from raw PPG signal, captured from an Android phone.

This paper presents a simple method to indirectly estimate the range of certain important electrocardiogram (ECG) parameters using photoplethysmography (PPG). The proposed method, termed as PhotoECG, extracts a set of time and frequency domain features from fingertip PPG signal. A feature selection algorithm utilizing the concept of Maximal Information Coefficient (MIC) is presented to rank the PPG features according to their relevance to create training models for different ECG parameters. The proposed method yields above 90% accuracy in estimating ECG parameters on a benchmark hospital dataset having clean PPG signal. The same method results an average of 80% accuracy on noisy PPG signal captured by iPhone, indicating its feasibility to create phone applications for preventive ECG monitoring at home.

In this paper we propose to demonstrate a smart phone application, that estimates human blood pressure (BP) values from photoplethysmography (PPG) signal using Windkessel model. PPG signal is extracted from a video sequence of a user's index fingertip, acquired using smart phone camera. A set of time domain PPG features are used to estimate different lumped parameters of Windkessel model to simulate arterial BP. Under most of the cases, the application estimates systolic and diastolic BP values, within a range of ±10% of clinical measurement.

Regular ECG check up is a good practice for cardiac patients as well as elderly people. In this paper we propose a low cost methodology to coarsely estimate the range of some important parameters of ECG using Photoplethysmography (PPG). PPG is easy to measure (even with a smart phone) and strongly related to human cardio-vascular system. The proposed methodology extracts a set of time domain features from PPG signal. A statistical analysis is performed to select the most relevant set of PPG features for the ECG parameters. Training model for the ECG parameters are created based on those selected features. Both artificial neural network and support vector machine based supervised learning approach is used for performance comparison. Experimental results, performed on benchmark dataset shows that good accuracy in the estimation of ECG parameters can be achieved in our proposed methodology. Results also show that the overall performance improves in using feature selection technique rather than using all the PPG features for classification.

Unobtrusive people identification based on gait recognition is steadily growing attention in modern video surveillance systems. In this paper we propose a two tier architecture for gait based people identification. The proposed architecture employs multiple Kinects working together as an integrated system to broaden the coverage area of surveillance. An intelligent triggering mechanism is presented for switching on and off the infrared (IR) cameras of different Kinects for optimum resource utilization. For efficient utilization of network bandwidth, the skeleton data from all the Kinects' edge devices (controllers) are compressed and sent to a backend server for people identification. Experimental results show that the triggering mechanism is capable of reducing the average power consumption of the Kinect controllers by more than 40 percent. Results also indicate that the person recognition accuracy does not degrade with the usage of the data compression, which reduces the data rate by 4 to 6 times while preserving their statistical properties.

Regular monitoring of certain vital parameters like heart-rate (HR), blood pressure (BP), Electrocardiogram (ECG) are the basic needs for elderly people and patients with chronic diseases residing at home. In this demo, authors would like to demonstrate the possibility of estimating BP levels and certain ECG parameters using the PPG signals captured from smart phones. The work includes mainly three components -- (i) robust PPG signal acquisition, (ii) estimation of BP levels (low, medium, high) from PPG signals and (iii) estimation of PR, RR, QRS and QT intervals of ECG parameters from PPG signals. Initially certain time domain features are extracted from PPG, which are used to create training models for various BP levels and ECG parameters. The approach is tested on two benchmark hospital datasets from (i) University of Queensland and (ii) Capnobase TBME RR dataset and one dataset captured from smart phones. Results indicate that the estimation accuracy is above 75% and sometimes above 95% if the height, weight and age information are considered.