The detection and localization of radiation sources using low-cost, mobile detectors is a challenging application, necessitating new research into sensing devices and detection algorithms. While new sensing that employs small detectors for detecting γ -rays has emerged, the decreased sensitivity of the sensor makes it challenging to maintain reliability compared to larger detectors. Machine learning could be a viable method for enhancing sensitivity by classifying background radiation spectra from anomalous spectra, but this approach can struggle to identify novel radioactive sources or identify sources in dynamic background environments. To address these challenges, we propose the use of adversarial auto encoders (AAEs) for anomaly detection in radiation sensing systems. With the use of our AAE architecture, we eliminate the need for obtaining examples of radiation anomalies for training data and increase the resilience of the sensing when the background radiation is dynamic. We evaluate the system in various contexts using a custom designed detector, showing the AAE model generalizes to various locations and radiation sources. We also show a real-time field test with the detection system in both handheld and drone mounted testing. • Our model improves detection of γ -ray anomalies on cesium iodide detectors. • These improvements extend to simulated sodium iodide detectors. • We demonstrate real-time detection capabilities in both lab and field environments. • These results are competitive with other unsupervised methods.

Autonomous vehicles (AVs) use complex suites of sensors to understand the surrounding environment and inform decision making systems which ensure the efficiency and safety of their operation. Light Detection and Ranging (LiDAR) sensors are an important part of the perception subsystems of many AVs, and are responsible for identifying obstacles to prevent collisions. This critical function makes LiDAR sensors a prime target for malicious attacks such as object hiding attacks in which an attacker uses a laser to spoof a LiDAR point cloud to cause an object to be “hidden” from the AV. However, despite many AVs today having multiple LiDAR sensors with overlapping fields of view, LiDAR spoofing attacks described in existing literature test only on single LiDAR systems. We hypothesize that sensor redundancy can effectively "fill-in" spoofed regions if one of the sensors is attacked, providing some inherent resilience to object hiding attacks. In this work, we evaluate the effectiveness of two object hiding attacks, the Object Removal Attack (ORA) and the Physical Removal Attack (PRA), on an AV digital twin with one, two, and three LiDAR configurations. We report up to an 83% reduction in attack success rate when using multi-LiDAR configurations for both ORA and PRA when hiding vehicles and up to a 100% and 75% reduction for ORA and PRA, respectively, when hiding pedestrians.

In education, analyzing student oral reading is critical to reading comprehension, vocabulary development, and fluency. Automating these assessments is a crucial part of language research– however, researchers often use a separate model for each assessment. Transferring knowledge between assessments can make a model more reliable and robust. In this work, we introduce a unified model comprising a contrastive self-supervised embedding model and a transfer learning suffix network. The embedding model is trained on oral reading fluency data, generating student voice embeddings. Subsequently, the transfer learning network employs these embeddings as features to estimate oral reading prosody scores. Our model demonstrates high agreement with human raters, with a reliability of 0.55 (78% accuracy) on unseen passages compared to 0.63 between humans. This generalization suggests that the model can provide robust representations across various oral reading assessment tasks.

We introduce a differentiable statistical moment aggregation layer, enabling networks to learn the optimal method of statistical moment pooling for automatic modulation classification. Statistical pooling, a cornerstone of convolutional networks, consolidates activations into fixed-length representations. Traditionally, this entails mean, variance, and higher-ordered statistics pooling defined as fixed hyperparameters. By enabling the statistics layer to become differentiable, networks are able to optimize the method of statistical aggregations, transcending predefined hyperparameters. With our approach, the statistical moment order is differentiable. Our results demonstrate learned statistical moments are able to outperform fixed-moments—improving modulation classification performance of a time-domain signal.1

Oral reading fluency assessment is a process where a student reads a passage aloud and is scored against words read correctly by a human listener. Current automatic reading fluency systems match these words read using speech recognition models trained with clean speech data from native adult speakers. This mismatch in training and deployment, compounded by numerous background noises from the classroom, means that student speech is often not correctly recognized. This paper describes a deep learning model that employs text-to-speech and contrastive learning to create acoustic word embeddings of student speech. This embedding is trained with unlabeled data of students reading known passages. Our model then uses sub-sequence matching in the acoustic embedding space to estimate words read correctly per minute, a common criterion in oral reading fluency. Our model’s words read correctly per minute is significantly closer to human listeners compared to systems that use automatic speech recognition only, reducing error of words correct per minute from 15.1 to 8.4, on average.

Virtualization and cloud computing have become critical parts of modern enterprise computing infrastructure. One of the benefits of using cloud infrastructure over in-house computing infrastructure is the offloading of security responsibilities. By hosting one’s services on the cloud, the responsibility for the security of the infrastructure is transferred to a trusted third party. As such, security of customer data in cloud environments is of critical importance. Side channels and covert channels have proven to be dangerous avenues for the leakage of sensitive information from computing systems. In this work, we propose and perform two experiments to investigate side and covert channel possibilities in virtual, enterprise environments. The first experiment is centered around the use of sensor data available via Intelligent Platform Management Interface, an open standard for out-of-band management often shipped with enterprise-level servers. We show how power-related sensors available with minimal user privilege over IPMI can be correlated with the levels of CPU stress of a virtual machine on a server. This leads to our second experiment, in which we demonstrate a power analysis approach for establishing a covert channel for the exfiltration of data from a network-isolated virtual machine on a server rack. By applying the concept of power analysis more broadly to the power consumption of an entire server rack, rather than individual hardware components, we find that basic patterns in system load can be clearly identified using signal processing techniques, demonstrating the potential for establishing a covert channel.

In this investigation we propose new machine learning methods for automated scoring models that predict the vocabulary acquisition in science and social studies of second grade English language learners, based upon free-form spoken responses. We evaluate performance on an existing dataset and use transfer learning from a large pre-trained language model, reporting the influence of various objective function designs and the input-convex network design. In particular, we find that combining objective functions with varying properties, such as distance among scores, greatly improves the model reliability compared to human raters. Our models extend the current state of the art performance for assessing word definition tasks and sentence usage tasks in science and social studies, achieving excellent quadratic weighted kappa scores compared with human raters. However, human-human agreement still surpasses model-human agreement, leaving room for future improvement. Even so, our work highlights the scalability of automated vocabulary assessment of free-form spoken language tasks in early grades.

Sentinel-1 C-band radar backscatter satellite images provide a repeating sequence of fine-resolution (10 m) observations that can be used for a number of applications, but the 12-day interval between satellite observations is too infrequent for many applications, such as measuring moisture dynamics. For a variety of applications, moisture information is demanded at high temporal frequency and fine spatial resolution over large areas. Machine learning approaches have been used to predict higher spatial resolutions than the original satellite images, but little effort has been made to increase the temporal resolution of acquired backscatter images. This study extends machine learning approaches to infer fine-resolution backscatter between observations relying on auxiliary data observations, including elevation and daily gridded weather. Several variations of multimodal fully convolutional neural network (CNN) architectures, problem setup, and training methods are explored for a predominantly rural area in southwest Oklahoma near the transition between humid subtropical and semiarid climates. The training area lies in the overlap zone for adjacent Sentinel-1 satellite tracks, allowing for training with several different temporal offsets. We find that the U-shaped fully convolutional neural NETwork (UNET) architecture produced the most accurate and robust estimated backscatter patterns, with superior prediction compared to a prior observation baseline in nearly all cases investigated when geography was included in the training data. This superior performance also generalized to nearby areas when training data for a given geography was not available, where 86% of predictions performed superior compared to a prior observation baseline.

Automated prosody classification in the context of oral reading fluency is a critical area for the objective evaluation of students’ reading proficiency. In this work, we present the largest dataset to date in this domain. It includes spoken phrases from over 1,300 students assessed by multiple trained raters. Moreover, we investigate the usage of X-Vectors and two variations thereof that incorporate weighted attention in classifying prosody correctness. We also evaluate the usage of quadratic weighted kappa loss to better accommodate the inter-rater differences in the dataset. Results indicate improved performance over baseline convolutional and current state-of-the-art models, with prosodic correctness accuracy of 86.4%.

Using spectral multiplication to compute convolution in neural networks has been investigated by a number of researchers because of its potential in speeding up computations for large images. However, previous methods require the learning of arbitrarily large convolution filters in the spectral domain, causing two untenable problems: an explosion in the number of trainable parameters per filter and an inability to reuse filters across images of differing sizes. To address this, we propose the usage of spectral parametric functions to represent massive spectral domain filters with only a few trainable parameters. Our empirical analysis suggests that the proposed functions maintain the benefits of arbitrarily large filters (such as improved rate of convergence in training, accuracy, and stability) while relying on significantly fewer trainable parameters.

Empirical results have shown that deep neural networks achieve superior performance in the application of Knowledge Tracing. However, the design of recurrent cells like long short term memory (LSTM) cells or gated recurrent units (GRU) is influenced largely by applications in natural language processing. They were proposed and evaluated in the context of sequence to sequence modeling, like machine translation. Even though the LSTM cell works well for knowledge tracing, it is unknown if its architecture is ideally suited for knowledge tracing. Despite the fact that there are several recurrent neural network based architectures proposed for knowledge tracing, the methodologies rely on empirical observations and trial and error, which may not be efficient or scalable. In this study, we investigate using reinforcement learning for the automatic design of recurrent neural network cells for knowledge tracing, showing improved performance compared to the LSTM cell. We also discuss a potential method for model regularization using neural architecture search.

Abstract The tremendous success of deep learning for imaging applications has resulted in numerous beneficial advances. Unfortunately, this success has also been a catalyst for malicious uses such as photo-realistic face swapping of parties without consent. In this study, we use deep transfer learning for face swapping detection, showing true positive rates greater than 96% with very few false alarms. Distinguished from existing methods that only provide detection accuracy, we also provide uncertainty for each prediction, which is critical for trust in the deployment of such detection systems. Moreover, we provide a comparison to human subjects. To capture human recognition performance, we build a website to collect pairwise comparisons of images from human subjects. Based on these comparisons, we infer a consensus ranking from the image perceived as most real to the image perceived as most fake. Overall, the results show the effectiveness of our method. As part of this study, we create a novel dataset that is, to the best of our knowledge, the largest swapped face dataset created using still images. This dataset will be available for academic research use per request. Our goal of this study is to inspire more research in the field of image forensics through the creation of a dataset and initial analysis.

Linear adaptive systems are a well-known staple in numerous signal processing applications. Recently, significant activity and performance gains have been achieved in multilayer neural networks for deep learning applied to practical data processing applications. In this paper, we describe the important relationships and significant differences between the procedures and methods used in linear adaptive systems and those used in multilayer neural networks for deep learning tasks. Input-output structures, cost functions and training criteria, adaptive algorithms, and data processing and optimization strategies are considered. It is the hope of the authors that this discussion will spur further crossover between the two fields, and in particular allow knowledge to be shared and further progress to be made.

Arterial oxygen saturation ([Formula: see text]) is an indicator of how much oxygen is carried by hemoglobin in the blood. Having enough oxygen is vital for the functioning of cells in the human body. Measurement of [Formula: see text] is typically estimated with a pulse oximeter, but recent works have investigated how smartphone cameras can be used to infer [Formula: see text]. In this paper, we propose methods for the measurement of [Formula: see text] with a smartphone using convolutional neural networks and preprocessing steps to better guard against motion artifacts. To evaluate this methodology, we conducted a breath-holding study involving 39 participants. We compare the results using two different mobile phones. We compare our model with the ratio-of-ratios model that is widely used in pulse oximeter applications, showing that our system has significantly lower mean absolute error (2.02%) than a medical pulse oximeter.

Cost and accessibility have impeded the adoption of spirometers (devices that measure lung function) outside clinical settings, especially in low-resource environments. Prior work, called SpiroSmart, used a smartphone's built-in microphone as a spirometer. However, individuals in low- or middle-income countries do not typically have access to the latest smartphones. In this paper, we investigate how spirometry can be performed from any phone-using the standard telephony voice channel to transmit the sound of the spirometry effort. We also investigate how using a 3D printed vortex whistle can affect the accuracy of common spirometry measures and mitigate usability challenges. Our system, coined SpiroCall, was evaluated with 50 participants against two gold standard medical spirometers. We conclude that SpiroCall has an acceptable mean error with or without a whistle for performing spirometry, and advantages of each are discussed.

Spirometry is the gold standard for managing and diagnosing obstructive lung diseases. Clinical spirometers, however, are expensive and have limited portability. Vortex whistles have shown promise as a potential substitute for clinical spirometers. While vortex whistles are low-cost and are highly portable, only a subset of common spirometry measurements can be measured reliably. Moreover, no research studies have evaluated characteristics of human interaction with vortex whistles, such as maneuver learnability and mental effort. We present a modified 3D-printed vortex whistle design that enables estimation of spirometry measures not previously attainable with traditional vortex whistles. We evaluate the whistle using a pulmonary waveform generator (a commercial standard) and map parameters of the whistle construction to spirometry test endpoints. Through a human subjects trial we evaluate how to personalize whistle parameters for different subjects and assess cognitive workload while using a vortex whistle. We show that, with personalization, vortex whistles are as effective as clinical spirometers for identifying moderate airway obstruction and require similar cognitive load to use.

Cognitive load refers to the amount of informationa person can process or hold in working memory. Historically, the psychology community has estimated this quantity objectively by monitoring the involuntary dilations and constrictions of the pupil using medical grade equipment known as pupillometers. At the same time, researchers in the HCI and Ubi Comp communities have hypothesized how cognitive load sensing might be integrated into context aware computing systems, but limitations of sensing cognitive load ubiquitously and reliably prevent the mass integration of such a technology. Our system, Pupil Ware-M, seeks to begin bridging this sensing gap. We build upon a recent platform, Pupil Ware, which measures a user's sub-millimeter pupil dilation from an unmodified camera. We update the Pupil Ware sensing system with a calibration protocol that brings pupillary responses of a diverse range of people and lighting conditions onto a single 0.0-1.0 scale called Cog Point. Furthermore, we update and optimize the algorithms employed to run in real time from a smartphone. We validate the calibration process using eight users in a controlled experiment where cognitive load is simple to determine from its situational context. Discussion of future work and remaining challenges is then described.

Electricity and appliance usage information can often reveal the nature of human activities in a home. For instance, sensing the use of vacuum cleaner, a microwave oven, and kitchen appliances can give insights into a person's current activities. Instead of putting a sensor on each appliance, our technique is based on the idea that appliance usage can be sensed by their manifestations in an environment's existing electrical infrastructure. Prior approaches using this technique could only detect an appliance's on-off states; that is, they only sense “what” is being used, but not “how” it is used. In this paper, we introduce DOSE, a significant advancement for inferring operating states of electronic devices from a single sensing point in a home. When an electronic device is in operation, it generates time-varying Electromagnetic Interference (EMI) based upon its operating states (e.g., vacuuming on a rug vs. hardwood floor). This EMI noise is coupled to the power line and can be picked up from a single sensing hardware attached to the wall outlet in a house. Unlike prior data-driven approaches, we employ domain knowledge of the device's circuitry for semi-supervised model training to avoid tedious labeling process. We evaluated DOSE in a residential house for 2 months and found that operating states for 16 appliances could be estimated with an average accuracy of 93.8%. These fine-grained electrical characteristics affords rich feature sets of electrical events and have the potential to support various applications such as in-home activity inference, energy disaggregation and device failure detection.

Health sensing through smartphones has received considerable attention in recent years because of the devices' ubiquity and promise to lower the barrier for tracking medical conditions. In this paper, we focus on using smartphones to monitor newborn jaundice, which manifests as a yellow discoloration of the skin. Although a degree of jaundice is common in healthy newborns, early detection of extreme jaundice is essential to prevent permanent brain damage or death. Current detection techniques, however, require clinical tests with blood samples or other specialized equipment. Consequently, newborns often depend on visual assessments of their skin color at home, which is known to be unreliable. To this end, we present BiliCam, a low-cost system that uses smartphone cameras to assess newborn jaundice. We evaluated BiliCam on 100 newborns, yielding a 0.85 rank order correlation with the gold standard blood test. We also discuss usability challenges and design solutions to make the system practical.

Mobile and embedded electronics are pervasive in today's environment. As such, it is necessary to have a natural and intuitive way for users to indicate the intent to connect to these devices from a distance. We present DopLink, an ultrasonic-based device selection approach. It utilizes the already embedded audio hardware in smart devices to determine if a particular device is being pointed at by another device (i.e., the user waves their mobile phone at a target in a pointing motion). We evaluate the accuracy of DopLink in a controlled user study, showing that, within 3 meters, it has an average accuracy of 95% for device selection and 97% for finding relative device position. Finally, we show three applications of DopLink: rapid device pairing, home automation, and multi-display synchronization.

Vibrotactile devices suffer from poor energy efficiency, arising from a mismatch between the device and the impedance of the human skin. This results in over-sized actuators and excessive power consumption, and prevents development of more sophisticated, miniaturized and low-power mobile tactile devices. In this paper, we present the experimental evaluation of a vibrotactile system designed to match the impedance of the skin to the impedance of the actuator. This system is able to quadruple the motion of the skin without increasing power consumption, and produce sensations equivalent to a standard system while consuming 1/2 of the power. By greatly reducing the size and power constraints of vibrotactile actuators, this technology offers a means to realize more sophisticated, smaller haptic devices for the user interface community.

Few means currently exist for home occupants to learn about their water consumption: e.g., where water use occurs, whether such use is excessive and what steps can be taken to conserve. Emerging water sensing systems, however, can provide detailed usage data at the level of individual water fixtures (i.e., disaggregated usage data). In this paper, we perform formative evaluations of two sets of novel eco-feedback displays that take advantage of this disaggregated data. The first display set isolates and examines specific elements of an eco-feedback design space such as data and time granularity. Displays in the second set act as design probes to elicit reactions about competition, privacy, and integration into domestic space. The displays were evaluated via an online survey of 651 North American respondents and in-home, semi-structured interviews with 10 families (20 adults). Our findings are relevant not only to the design of future water eco-feedback systems but also for other types of consumption (e.g., electricity and gas).

Home spirometry is gaining acceptance in the medical community because of its ability to detect pulmonary exacerbations and improve outcomes of chronic lung ailments. However, cost and usability are significant barriers to its widespread adoption. To this end, we present SpiroSmart, a low-cost mobile phone application that performs spirometry sensing using the built-in microphone. We evaluate SpiroSmart on 52 subjects, showing that the mean error when compared to a clinical spirometer is 5.1% for common measures of lung function. Finally, we show that pulmonologists can use SpiroSmart to diagnose varying degrees of obstructive lung ailments.

We present HeatWave, a system that uses digital thermal imaging cameras to detect, track, and support user interaction on arbitrary surfaces. Thermal sensing has had limited examination in the HCI research community and is generally under-explored outside of law enforcement and energy auditing applications. We examine the role of thermal imaging as a new sensing solution for enhancing user surface interaction. In particular, we demonstrate how thermal imaging in combination with existing computer vision techniques can make segmentation and detection of routine interaction techniques possible in real-time, and can be used to complement or simplify algorithms for traditional RGB and depth cameras. Example interactions include (1) distinguishing hovering above a surface from touch events, (2) shape-based gestures similar to ink strokes, (3) pressure based gestures, and (4) multi-finger gestures. We close by discussing the practicality of thermal sensing for naturalistic user interaction and opportunities for future work.

Audio-based cough detection has become more pervasive in recent years because of its utility in evaluating treatments and the potential to impact the quality of life for individuals with chronic cough. We critically examine the current state of the art in cough detection, concluding that existing approaches expose private audio recordings of users and bystanders. We present a novel algorithm for detecting coughs from the audio stream of a mobile phone. Our system allows cough sounds to be reconstructed from the feature set, but prevents speech from being reconstructed intelligibly. We evaluate our algorithm on data collected in the wild and report an average true positive rate of 92% and false positive rate of 0.5%. We also present the results of two psychoacoustic experiments which characterize the tradeoff between the fidelity of reconstructed cough sounds and the intelligibility of reconstructed speech.

We present WATTR, a novel self-powered water activity sensor that utilizes residential water pressure impulses as both a powering and sensing source. Consisting of a power harvesting circuit, piezoelectric sensor, ultra-low-power 16-bit microcontroller, 16-bit analog-to-digital converter (ADC), and a 433 MHz wireless transmitter, WATTR is capable of sampling home water pressure at 33 Hz and transmitting over 3 m when any water fixture in the home is opened or closed. WATTR provides an alternative sensing solution to the power intensive Bluetooth-based sensor used in the HydroSense project by Froehlich et al. [2] for single-point whole-home water usage. We demonstrate WATTR as a viable self-powered sensor capable of monitoring and transmitting water usage data without the use of a battery. Unlike other water-based power harvesters, WATTR does not waste water to power itself. We discuss the design, implementation, and experimental verification of the WATTR device.

Recent work has examined infrastructure-mediated sensing as a practical, low-cost, and unobtrusive approach to sensing human activity in the physical world. This approach is based on the idea that human activities (e.g., running a dishwasher, turning on a reading light, or walking through a doorway) can be sensed by their manifestations in an environment's existing infrastructures (e.g., a home's water, electrical, and HVAC infrastructures). This paper presents HydroSense, a low-cost and easily-installed single-point sensor of pressure within a home's water infrastructure. HydroSense supports both identification of activity at individual water fixtures within a home (e.g., a particular toilet, a kitchen sink, a particular shower) as well as estimation of the amount of water being used at each fixture. We evaluate our approach using data collected in ten homes. Our algorithms successfully identify fixture events with 97.9% aggregate accuracy and can estimate water usage with error rates that are comparable to empirical studies of traditional utility-supplied water meters. Our results both validate our approach and provide a basis for future improvements.

Facial feature tracking for model-based coding has evolved over the past decades. Of particular interest is its application in very low bit rate coding in which optimization is used to analyze head and shoulder sequences. We present the results of a computational experiment in which we apply a combination of non-dominated sorting genetic algorithm (NSGA-II) and a deterministic search to find optimal facial animation parameters at many bandwidths, simultaneously. As objective functions are concerned, peak signal-to-noise ratio is chosen to be maximized while the total number of facial animation parameters is chosen to be minimized. Particularly, the algorithm is tested for efficiency and reliability. The results show that the overall methodology works effectively, but that a better error assessment function is needed.

This paper presents the results of a computational experiment designed to investigate the extent to which metrics of image fidelity can be improved through knowledge of where humans tend to fixate in images. Five common metrics of image fidelity were augmented using two sets of fixation data, one set obtained under task-free viewing conditions and another set obtained when viewers were asked to judge image quality. The augmented metrics were then compared to subjective ratings of the images. The results show that most metrics can be improved using eye fixation data, but a greater improvement was found using fixations obtained in the task-free condition (task-free viewing).