While NLP research has made strides in conversational tasks, many approaches focus on single-turn responses with well-defined objectives or evaluation criteria. In contrast, coaching presents unique challenges with initially undefined goals that evolve through multi-turn interactions, subjective evaluation criteria, mixed-initiative dialogue. In this work, we describe and implement five multi-turn coaching agents that exhibit distinct conversational styles, and evaluate them through a user study, collecting first-person feedback on 155 conversations. We find that users highly value core functionality, and that stylistic components in absence of core components are viewed negatively. By comparing user feedback with third-person evaluations from health experts and an LM, we reveal significant misalignment across evaluation approaches. Our findings provide insights into design and evaluation of conversational coaching agents and contribute toward improving human-centered NLP applications.

Life cycle assessment (LCA) is a methodology for holistically measuring the environmental impact of a product from initial manufacturing to end-of-life disposal.However, the extent to which LCA informs the design of computing devices remains unclear.To understand how this information is collected and applied, we interviewed 17 industry professionals with experience in LCA or electronics design, systematically coded the interviews, and investigated common themes.These themes highlight the challenge of LCA data collection and reveal distributed decision-making processes where responsibility for sustainable design choices-and their associated costs-is often ambiguous.Our analysis identifes opportunities for HCI technologies to support LCA computation and its integration into the design process to facilitate sustainability-oriented decision-making.While this work provides a nuanced discussion about sustainable design in the information and communication technologies (ICT) hardware industry, we hope our insights will also be valuable to other sectors.

Wearable sensors have become ubiquitous thanks to a variety of health tracking features. The resulting continuous and longitudinal measurements from everyday life generate large volumes of data. However, making sense of these observations for scientific and actionable insights is non-trivial. Inspired by the empirical success of generative modeling, where large neural networks learn powerful representations from vast amounts of text, image, video, or audio data, we investigate the scaling properties of wearable sensor foundation models across compute, data, and model size. Using a dataset of up to 40 million hours of in-situ heart rate, heart rate variability, accelerometer, electrodermal activity, skin temperature, and altimeter per-minute data from over 165,000 people, we create LSM, a multimodal foundation model built on the largest wearable-signals dataset with the most extensive range of sensor modalities to date. Our results establish the scaling laws of LSM for tasks such as imputation, interpolation and extrapolation across both time and sensor modalities. Moreover, we highlight how LSM enables sample-efficient downstream learning for tasks including exercise and activity recognition.

Language models (LMs) are increasingly being deployed to perform autonomous data analyses. However, their data awareness—the ability to recognize, reason over, and appropriately handle data artifacts such as missing values, outliers, and logical inconsistencies—remains underexplored. These artifacts are especially common in real-world tabular data and, if mishandled, can significantly compromise the validity of analytical conclusions. To address this gap, we present RADAR, a benchmark for systematically evaluating data-aware reasoning on tabular data. We develop a framework to simulate data artifacts via programmatic perturbations to enable targeted evaluation of model behavior. RADAR comprises 2,980 table-query pairs, grounded in real-world data spanning 9 domains and 5 data artifact types. In addition to evaluating artifact handling, RADAR systematically varies table size to study how reasoning performance holds when increasing table size. Our evaluation reveals that, despite decent performance on tables without data artifacts, frontier models degrade significantly when data artifacts are introduced, exposing critical gaps in their capacity for robust, data-aware analysis. Designed to be flexible and extensible, RADAR supports diverse perturbation types and controllable table sizes, offering a valuable resource for advancing tabular reasoning.

We present SensorLM, a family of sensor-language foundation models that enable wearable sensor data understanding with natural language. Despite its pervasive nature, aligning and interpreting sensor data with language remains challenging due to the lack of paired, richly annotated sensor-text descriptions in uncurated, real-world wearable data. We introduce a hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data. This approach enabled the curation of the largest sensor-language dataset to date, comprising over 59.7 million hours of data from more than 103,000 people. Furthermore, SensorLM extends prominent multimodal pretraining architectures (e.g., CLIP, CoCa) and recovers them as specific variants within a generic architecture. Extensive experiments on real-world tasks in human activity analysis and healthcare verify the superior performance of SensorLM over state-of-the-art in zero-shot recognition, few-shot learning, and cross-modal retrieval. SensorLM also demonstrates intriguing capabilities including scaling behaviors, label efficiency, sensor captioning, and zero-shot generalization to unseen tasks. Code is available at https://github.com/Google-Health/consumer-health-research/tree/main/sensorlm.

As Extended Reality (XR) advances, a device has the potential to be used across contexts from immersive productivity at a desk to on-the-go, public scenarios. Existing input solutions lack the versatility to provide both high-throughput, mouse-grade input and subtle, ergonomic interaction. We introduce FlowRing, a novel ring-form device that combines microgestures with precise 2D mouse-like input on surfaces. FlowRing supports five microgestures for discreet interaction and 2D input for richer tasks, using an optical flow sensor, skin-contact microphone, and IMU at the base of the finger. In a study with 11 participants, FlowRing achieved 93.6% microgesture recognition accuracy across sessions and 85.2% across unseen users, rising to 90.1% with just four gesture set examples from a new user. A separate 2D Fitts’ law study demonstrated its effectiveness for continuous input on various surfaces. FlowRing emerges as a versatile, user-friendly solution for the future of interactive technology.

Crowdsourcing platforms have transformed distributed problem-solving, yet quality control remains a persistent challenge. Traditional quality control measures, such as prescreening workers and refining instructions, often focus solely on optimizing economic output. This paper explores just-in-time AI interventions to enhance both labeling quality and domain-specific knowledge among crowdworkers. We introduce LabelAId, an advanced inference model combining Programmatic Weak Supervision (PWS) with FT-Transformers to infer label correctness based on user behavior and domain knowledge. Our technical evaluation shows that our LabelAId pipeline consistently outperforms state-of-the-art ML baselines, improving mistake inference accuracy by 36.7% with 50 downstream samples. We then implemented LabelAId into Project Sidewalk, an open-source crowdsourcing platform for urban accessibility. A between-subjects study with 34 participants demonstrates that LabelAId significantly enhances label precision without compromising efficiency while also increasing labeler confidence. We discuss LabelAId’s success factors, limitations, and its generalizability to other crowdsourced science domains.

Integrating cameras into wireless smart rings has been challenging due to size and power constraints. We introduce IRIS, the first wireless vision-enabled smart ring system for smart home interactions. Equipped with a camera, Bluetooth radio, inertial measurement unit (IMU), and an onboard battery, IRIS meets the small size, weight, and power (SWaP) requirements for ring devices. IRIS is context-aware, adapting its gesture set to the detected device, and can last for 16-24 hours on a single charge. IRIS leverages the scene semantics to achieve instance-level device recognition. In a study involving 23 participants, IRIS consistently outpaced voice commands, with a higher proportion of participants expressing a preference for IRIS over voice commands regarding toggling a device’s state, granular control, and social acceptability. Our work pushes the boundary of what is possible with ring form-factor devices, addressing system challenges and opening up novel interaction capabilities.

We introduce a low-power communication method that lets smartwatches leverage existing electrocardiogram (ECG) hardware as a data communication interface. Our unique approach enables the connection of external, inexpensive, and low-power "add-on" sensors to the smartwatch, expanding its functionalities. These sensors cater to specialized user needs beyond those offered by pre-built sensor suites, at a fraction of the cost and power of traditional communication protocols, including Bluetooth Low Energy. To demonstrate the feasibility of our approach, we conduct a series of exploratory and evaluative tests to characterize the ECG interface as a communication channel on commercial smartwatches. We design a simple transmission scheme using commodity components, demonstrating cost and power benefits. Further, we build and test a suite of add-on sensors, including UV light, body temperature, buttons, and breath alcohol, all of which achieved testing objectives at low material cost and power usage. This research paves the way for personalized and user-centric wearables by offering a cost-effective solution to expand their functionalities.

Understanding of human visual perception has historically inspired the design of computer vision architectures. As an example, perception occurs at different scales both spatially and temporally, suggesting that the extraction of salient visual information may be made more effective by attending to specific features at varying scales. Visual changes in the body, due to physiological processes, also occur at varying scales and with modality-specific characteristic properties. Inspired by this, we present BigSmall, an efficient architecture for physiological and behavioral measurement. We present the first joint camera-based facial action, cardiac, and pulmonary measurement model. We propose a multi-branch network with wrapping temporal shift modules that yields efficiency gains and accuracy on par with task-optimized methods. We observe that fusing low-level features leads to suboptimal performance, but that fusing high level features enables efficiency gains with negligible losses in accuracy. We experimentally validate that BigSmall significantly reduces computational cost while achieving comparable results on multiple physiological measurement tasks simultaneously with a unified model.

Machine learning models for camera-based physiological measurement can have weak generalization due to a lack of representative training data. Body motion is one of the most significant sources of noise when attempting to recover the subtle cardiac pulse from a video. We explore motion transfer as a form of data augmentation to introduce motion variation while preserving physiological changes of interest. We adapt a neural video synthesis approach to augment videos for the task of remote photoplethysmography (rPPG) and study the effects of motion augmentation with respect to 1) the magnitude and 2) the type of motion. After training on motion-augmented versions of publicly available datasets, we demonstrate a 47% improvement over existing inter-dataset results using various state-of-the-art methods on the PURE dataset. We also present inter-dataset results on five benchmark datasets to show improvements of up to 79% using TS-CAN, a neural rPPG estimation method. Our findings illustrate the usefulness of motion transfer as a data augmentation technique for improving the generalization of models for camera-based physiological sensing. We release our code for using motion transfer as a data augmentation technique on three publicly available datasets, UBFC-rPPG, PURE, and SCAMPS, and models pre-trained on motion-augmented data here: https://motion-matters.github.io/

A computer vision system using low-resolution image sensors can provide intelligent services (e.g., activity recognition) but preserve unnecessary visual privacy information from the hardware level. However, preserving visual privacy and enabling accurate machine recognition have adversarial needs on image resolution. Modeling the trade-off of privacy preservation and machine recognition performance can guide future privacy-preserving computer vision systems using low-resolution image sensors. In this paper, using the at-home activity of daily livings (ADLs) as the scenario, we first obtained the most important visual privacy features through a user survey. Then we quantified and analyzed the effects of image resolution on human and machine recognition performance in activity recognition and privacy awareness tasks. We also investigated how modern image super-resolution techniques influence these effects. Based on the results, we proposed a method for modeling the trade-off of privacy preservation and activity recognition on low-resolution images.

Designing privacy-respecting and human-centric smart cities requires a careful investigation of people’s attitudes and concerns toward city-wide data collection scenarios. To capture a holistic view, we carried out this investigation in two phases. We first surfaced people’s understanding, concerns, and expectations toward smart city scenarios by conducting 21 semi-structured interviews with people in underserved communities. We complemented this in-depth qualitative study with a 348-participant online survey of the general population to quantify the significance of smart city factors (e.g., type of collected data) on attitudes and concerns. Depending on demographics, privacy and ethics were the two most common types of concerns among participants. We found the type of collected data to have the most and the retention time to have the least impact on participants’ perceptions and concerns about smart cities. We highlight key takeaways and recommendations for city stakeholders to consider when designing inclusive and protective smart cities.

We present Z-Ring, a wearable ring that enables gesture input, object detection, user identification, and interaction with passive user interface (UI) elements using a single sensing modality and a single point of instrumentation on the finger. Z-Ring uses active electrical field sensing to detect changes in the hand’s electrical impedance caused by finger motions or contact with external surfaces. We develop a diverse set of interactions and evaluate them with 21 users. We demonstrate: (1) Single- and two-handed gesture recognition with up to 93% accuracy (2) Tangible input with a set of passive touch UI elements, including buttons, a continuous 1D slider, and a continuous 2D trackpad with 91.8% accuracy, <4.4 cm MAE, and <4.1cm MAE, respectively (3) Object recognition across six household objects with 94.5% accuracy (4) User identification among 14 users with 99% accuracy. Z-Ring’s sensing methodology uses only a single co-located electrode pair for both receiving and sensing, lending itself well to future miniaturization for use in on-the-go scenarios.

We present MilliMobile: a first of its kind battery-free autonomous robot capable of operating on harvested solar and RF power. We challenge the conventional assumption that motion and actuation are beyond the capabilities of battery-free devices and demonstrate completely untethered autonomous operation in realistic indoor and outdoor lighting as well as RF power delivery scenarios. We show first that through miniaturizing a robot to gram scale, we can significantly reduce the energy required to move it. Second, we develop methods to produce intermittent motion by discharging a small capacitor (47--150 μF) to move a motor in discrete steps, enabling motion from as little as 50 μW of power or less. We further develop software defined techniques for maximizing power harvesting. MilliMobile operates in the optimal part of the charging curve by varying the charging time to achieve maximum speeds of up to 5.5 mm/s.

We present the first wireless earbud hardware that can perform hearing screening by detecting otoacoustic emissions. The conventional wisdom has been that detecting otoacoustic emissions, which are the faint sounds generated by the cochlea, requires sensitive and expensive acoustic hardware. Thus, medical devices for hearing screening cost thousands of dollars and are inaccessible in low and middle income countries. We show that by designing wireless ear-buds using low-cost acoustic hardware and combining them with wireless sensing algorithms, we can reliably identify otoacoustic emissions and perform hearing screening. Our algorithms combine frequency modulated chirps with wideband pulses emitted from a low-cost speaker to reliably separate otoacoustic emissions from in-ear reflections and echoes. We conducted a clinical study with 50 ears across two healthcare sites. Our study shows that the low-cost earbuds detect hearing loss with 100% sensitivity and 89.7% specificity, which is comparable to the performance of a $8000 medical device. By developing low-cost and open-source wearable technology, our work may help address global health inequities in hearing screening by democratizing these medical devices.

Camera-based physiological measurement is a fast growing field of computer vision. Remote photoplethysmography (rPPG) utilizes imaging devices (e.g., cameras) to measure the peripheral blood volume pulse (BVP) via photoplethysmography, and enables cardiac measurement via webcams and smartphones. However, the task is non-trivial with important pre-processing, modeling and post-processing steps required to obtain state-of-the-art results. Replication of results and benchmarking of new models is critical for scientific progress; however, as with many other applications of deep learning, reliable codebases are not easy to find or use. We present a comprehensive toolbox, rPPG-Toolbox, unsupervised and supervised rPPG models with support for public benchmark datasets, data augmentation and systematic evaluation: https://github.com/ubicomplab/rPPG-Toolbox.

Camera-based physiological measurement is a growing field with neural models providing state-of-the-art performance. Prior research has explored various "end-to-end" architectures; however these methods still require several preprocessing steps and are not able to run directly on mobile and edge devices. The operations are often non-trivial to implement, making replication and deployment difficult and can even have a higher computational budget than the "core" network itself. In this paper, we propose two novel and efficient neural models for camera-based physiological measurement called EfficientPhys that remove the need for face detection, segmentation, normalization, color space transformation or any other preprocessing steps. Using an input of raw video frames, our models achieve strong accuracy on three public datasets. We show that this is the case whether using a transformer or convolutional backbone. We further evaluate the latency of the proposed networks and show that our most lightweight network also achieves a 33% improvement in efficiency.

According to the World Health Organisation (WHO), 235 million people suffer from respiratory illnesses which causes four million deaths annually. Regular lung health monitoring can lead to prognoses about deteriorating lung health conditions. This article presents our system SpiroMask that retrofits a microphone in consumer-grade masks (N95 and cloth masks) for continuous lung health monitoring. We evaluate our approach on 48 participants (including 14 with lung health issues) and find that we can estimate parameters such as lung volume and respiration rate within the approved error range by the American Thoracic Society (ATS). Further, we show that our approach is robust to sensor placement inside the mask.

Face orientation can often indicate users’ intended interaction target. In this paper, we propose FaceOri, a novel face tracking technique based on acoustic ranging using earphones. FaceOri can leverage the speaker on a commodity device to emit an ultrasonic chirp, which is picked up by the set of microphones on the user’s earphone, and then processed to calculate the distance from each microphone to the device. These measurements are used to derive the user’s face orientation and distance with respect to the device. We conduct a ground truth comparison and user study to evaluate FaceOri’s performance. The results show that the system can determine whether the user orients to the device at a 93.5% accuracy within a 1.5 meters range. Furthermore, FaceOri can continuously track user’s head orientation with a median absolute error of 10.9 mm in the distance, 3.7° in yaw, and 5.8° in pitch. FaceOri can allow for convenient hands-free control of devices and produce more intelligent context-aware interactions.

We present ClearBuds, the first hardware and software system that utilizes a neural network to enhance speech streamed from two wireless earbuds. Real-time speech enhancement for wireless earbuds requires high-quality sound separation and background cancellation, operating in real-time and on a mobile phone. Clear-Buds bridges state-of-the-art deep learning for blind audio source separation and in-ear mobile systems by making two key technical contributions: 1) a new wireless earbud design capable of operating as a synchronized, binaural microphone array, and 2) a lightweight dual-channel speech enhancement neural network that runs on a mobile device. Our neural network has a novel cascaded architecture that combines a time-domain conventional neural network with a spectrogram-based frequency masking neural network to reduce the artifacts in the audio output. Results show that our wireless earbuds achieve a synchronization error less than 64 μs and our network has a runtime of 21.4 ms on an accompanying mobile phone. In-the-wild evaluation with eight users in previously unseen indoor and outdoor multipath scenarios demonstrates that our neural network generalizes to learn both spatial and acoustic cues to perform noise suppression and background speech removal. In a user-study with 37 participants who spent over 15.4 hours rating 1041 audio samples collected in-the-wild, our system achieves improved mean opinion score and background noise suppression.

We present ClearBuds, the first end-to-end hardware and software system that utilizes a neural network to enhance speech streamed from two wireless earbuds. Real-time speech enhancement for wireless earbuds requires high-quality sound separation and background cancellation, operating in real-time and on a mobile phone. Clear-Buds bridges state-of-the-art deep learning for blind audio source separation and in-ear mobile systems by making two key technical contributions: 1) a new wireless earbud design capable of operating as a synchronized, binaural microphone array, and 2) a lightweight dual-channel speech enhancement neural network that runs on a mobile device. Our demo will allow MobiSys attendees wear our earbuds, and experience noise suppression as they talk in a noisy environment. Companion video can be accessed using the link below:

Recent research has demonstrated the capability of behavior signals captured by smartphones and wearables for longitudinal behavior modeling. However, there is a lack of a comprehensive public dataset that serves as an open testbed for fair comparison among algorithms. Moreover, prior studies mainly evaluate algorithms using data from a single population within a short period, without measuring the cross-dataset generalizability of these algorithms. We present the first multi-year passive sensing datasets, containing over 700 user-years and 497 unique users’ data collected from mobile and wearable sensors, together with a wide range of well-being metrics. Our datasets can support multiple cross-dataset evaluations of behavior modeling algorithms’ generalizability across different users and years. As a starting point, we provide the benchmark results of 18 algorithms on the task of depression detection. Our results indicate that both prior depression detection algorithms and domain generalization techniques show potential but need further research to achieve adequate cross-dataset generalizability. We envision our multi-year datasets can support the ML community in developing generalizable longitudinal behavior modeling algorithms.

We present LuckyChirp, a contactless, passive, opportunistic respiratory tracking solution for commodity device using cascaded sonar modeling. Compared to conventional sonar methods that only solve the respiratory estimation problem (“what is the respiratory rate”), LuckyChirp also solves the additional respiratory detection problem (“is the human present and static enough for respiration sensing”). LuckyChirp uses a custom neural network on pulsed sonar’s wavelet transformed features to detect respiration. The classifier is then cascaded with a respiratory rate estimator. Such holistic design eliminates user friction of manually activating the system and enables passive respiration monitoring for all-day natural use. With Google Nest Hub and Pixel 4 as experimental devices, LuckyChirp achieves a mean absolute error of 0.48±0.98 and 1.07±1.67 breaths/min, respectively, for 20 users participating in a whole-night study. Compared to direct respiratory estimation without respiration classification, this is a ×6 (Nest Hub) and ×4 (Pixel) reduction in error.

Debugging printed circuit boards (PCBs) can be a time-consuming process, requiring frequent context switching between PCB design files (schematic and layout) and the physical PCB. To assist electrical engineers in debugging PCBs, we present ARDW, an augmented reality workbench consisting of a monitor interface featuring PCB design files, a projector-augmented workspace for PCBs, tracked test probes for selection and measurement, and a connected test instrument. The system supports common debugging workflows for augmented visualization on the physical PCB as well as augmented interaction with the tracked probes. We quantitatively and qualitatively evaluate the system with 10 electrical engineers from industry and academia, finding that ARDW speeds up board navigation and provides engineers with greater confidence in debugging. We discuss practical design considerations and paths for improvement to future systems. A video demo of the system may be accessed here: https://youtu.be/RbENbf5WIfc .

Debugging printed circuit boards (PCBs) requires frequent context switching and spatial pattern matching between software design files and physical boards. To reduce this overhead, we conduct a series of interviews with electrical engineers to understand their workflows, around which we design a set of AR interaction techniques, we call Augmented Silkscreen, to streamline identification, localization, annotation, and measurement tasks. We then run a set of remote user studies with illustrative video sketches and simulated PCB tasks to compare our interactions with current practices, finding that our techniques reduce completion times. Based on these quantitative results, as well as qualitative feedback from our participants, we offer design recommendations for the implementation of these interactions on a future, deployable AR system.

As wearable devices move toward the face (i.e. smart earbuds, glasses), there is an increasing need to facilitate intuitive interactions with these devices. Current sensing techniques can already detect many mouth-based gestures; however, users’ preferences of these gestures are not fully understood. In this paper, we investigate the design space and usability of mouth-based microgestures. We first conducted brainstorming sessions (N=16) and compiled an extensive set of 86 user-defined gestures. Then, with an online survey (N=50), we assessed the physical and mental demand of our gesture set and identified a subset of 14 gestures that can be performed easily and naturally. Finally, we conducted a remote Wizard-of-Oz usability study (N=11) mapping gestures to various daily smartphone operations under a sitting and walking context. From these studies, we develop a taxonomy for mouth gestures, finalize a practical gesture set for common applications, and provide design guidelines for future mouth-based gesture interactions.

QWERTY is the primary smartphone text input keyboard configuration. However, insertion and substitution errors caused by hand tremors, often experienced by users with Parkinson’s disease, can severely affect typing efficiency and user experience. In this paper, we investigated Parkinson’s users’ typing behavior on smartphones. In particular, we identified and compared the typing characteristics generated by users with and without Parkinson’s symptoms. We then proposed an elastic probabilistic model for input prediction. By incorporating both spatial and temporal features, this model generalized the classical statistical decoding algorithm to correct insertion, substitution and omission errors, while maintaining direct physical interpretation. User study results confirmed that the proposed algorithm outperformed baseline techniques: users reached 22.8 WPM typing speed with a significantly lower error rate and higher user-perceived performance and preference. We concluded that our method could effectively improve the text entry experience on smartphones for users with Parkinson’s disease.

Learned speech representations can drastically improve performance on tasks with limited labeled data. However, due to their size and complexity, learned representations have limited utility in mobile settings where run-time performance can be a significant bottleneck. In this work, we propose a class of lightweight non-semantic speech embedding models that run efficiently on mobile devices based on the recently proposed TRILL speech embedding. We combine novel architectural modifications with existing speed-up techniques to create embedding models that are fast enough to run in real-time on a mobile device and exhibit minimal performance degradation on a benchmark of non-semantic speech tasks. One such model (FRILL) is 32x faster on a Pixel 1 smartphone and 40% the size of TRILL, with an average decrease in accuracy of only 2%. To our knowledge, FRILL is the highest-quality non-semantic embedding designed for use on mobile devices. Furthermore, we demonstrate that these representations are useful for mobile health tasks such as non-speech human sounds detection and face-masked speech detection. Our models and code are publicly available.

Unpredictable ML model behavior on unseen data, especially in the health domain, raises serious concerns about its safety as repercussions for mistakes can be fatal. In this paper, we explore the feasibility of using state-of-the-art out-of-distribution detectors for reliable and trustworthy diagnostic predictions. We select publicly available deep learning models relating to various health conditions (e.g., skin cancer, lung sound, and Parkinson's disease) using various input data types (e.g., image, audio, and motion data). We demonstrate that these models show unreasonable predictions on out-of-distribution datasets. We show that Mahalanobis distance- and Gram matrices-based out-of-distribution detection methods are able to detect out-of-distribution data with high accuracy for the health models that operate on different modalities. We then translate the out-of-distribution score into a human interpretable \textsc{confidence score} to investigate its effect on the users' interaction with health ML applications. Our user study shows that the \textsc{confidence score} helped the participants only trust the results with a high score to make a medical decision and disregard results with a low score. Through this work, we demonstrate that dataset shift is a critical piece of information for high-stake ML applications, such as medical diagnosis and healthcare, to provide reliable and trustworthy predictions to the users.

Sleep is critical to human function, mediating factors like memory, mood, energy, and alertness; therefore, it is commonly conjectured that a good night’s sleep is important for job performance. However, both real-world sleep behavior and job performance are difficult to measure at scale. In this work, we demonstrate that people’s everyday interactions with online mobile apps can reveal insights into their job performance in real-world contexts. We present an observational study in which we objectively tracked the sleep behavior and job performance of salespeople (N = 15) and athletes (N = 19) for 18 months, leveraging a mattress sensor and online mobile app to conduct the largest study of this kind to date. We first demonstrate that cumulative sleep measures are significantly correlated with job performance metrics, showing that an hour of daily sleep loss for a week was associated with a 9.0% average reduction in contracts established for salespeople and a 9.5% average reduction in game grade for the athletes. We then investigate the utility of online app interaction time as a passively collectible and scalable performance indicator. We show that app interaction time is correlated with the job performance of the athletes, but not the salespeople. To support that our app-based performance indicator truly captures meaningful variation in psychomotor function as it relates to sleep and is robust against potential confounds, we conducted a second study to evaluate the relationship between sleep behavior and app interaction time in a cohort of 274 participants. Using a generalized additive model to control for per-participant random effects, we demonstrate that participants who lost one hour of daily sleep for a week exhibited average app interaction times that were 5.0% slower. We also find that app interaction time exhibits meaningful chronobiologically consistent correlations with sleep history, time awake, and circadian rhythms. The findings from this work reveal an opportunity for online app developers to generate new insights regarding cognition and productivity.

Haptic feedback can significantly enhance the realism and immersiveness of virtual reality (VR) systems. In this paper, we propose MoveVR, a technique that enables realistic, multiform force feedback in VR leveraging commonplace cleaning robots. MoveVR can generate tension, resistance, impact and material rigidity force feedback with multiple levels of force intensity and directions. This is achieved by changing the robot's moving speed, rotation, position as well as the carried proxies. We demonstrated the feasibility and effectiveness of MoveVR through interactive VR gaming. In our quantitative and qualitative evaluation studies, participants found that MoveVR provides more realistic and enjoyable user experience when compared to commercially available haptic solutions such as vibrotactile haptic systems.

Current automatic cough counting systems can determine how many coughs are present in an audio recording. However, they cannot determine who produced the cough. This limits their usefulness as most systems are deployed in locations with multiple people (i.e., a smart home device in a four-person home). Previous models trained solely on speech performed reasonably well on forced coughs [1]. By incorporating coughs into the training data, the model performance should improve. However, since limited natural cough data exists, training on coughs can lead to model overfitting. In this work, we overcome this problem by using multitask learning, where the second task is speaker verification. Our model achieves 82.15% classification accuracy amongst four users on a natural, in-the-wild cough dataset, outperforming human evaluators on average by 9.82%.

Rapid diagnostic tests (RDTs) provide point-of-care medical diagnosis without sophisticated laboratory equipment, making them especially useful for community health workers (CHWs). Because the procedure for completing a malaria RDT is error-prone, CHWs are often asked to carry completed RDTs back to their supervisors. Doing so makes RDTs susceptible to deterioration and introduces inefficiencies in the CHWs' workflow. In this work, we propose a smartphone-based RDT capture app, RDTScan, that facilitates the collection of high-quality RDT images to support CHWs in the field. RDTScan does not require an external adapter to control the image capture environment, but instead provides real-time guidance using image processing to obtain the best image possible. During our evaluation study, we found that RDTScan had 98.1% sensitivity and 99.7% specificity against visual inspection of the RDTs. RDTScan helped CHWs capture high-quality RDT images within 18 seconds while enabling a better RDT workflow.

Gaze tracking is an essential component of next generation displays for virtual reality and augmented reality applications. Traditional camera-based gaze trackers used in next generation displays are known to be lacking in one or multiple of the following metrics: power consumption, cost, computational complexity, estimation accuracy, latency, and form-factor. We propose the use of discrete photodiodes and light-emitting diodes (LEDs) as an alternative to traditional camera-based gaze tracking approaches while taking all of these metrics into consideration. We begin by developing a rendering-based simulation framework for understanding the relationship between light sources and a virtual model eyeball. Findings from this framework are used for the placement of LEDs and photodiodes. Our first prototype uses a neural network to obtain an average error rate of 2.67° at 400 Hz while demanding only 16 mW. By simplifying the implementation to using only LEDs, duplexed as light transceivers, and more minimal machine learning model, namely a light-weight supervised Gaussian process regression algorithm, we show that our second prototype is capable of an average error rate of 1.57° at 250 Hz using 800 mW.

Telehealth and remote health monitoring have become increasingly important during the SARS-CoV-2 pandemic and it is widely expected that this will have a lasting impact on healthcare practices. These tools can help reduce the risk of exposing patients and medical staff to infection, make healthcare services more accessible, and allow providers to see more patients. However, objective measurement of vital signs is challenging without direct contact with a patient. We present a video-based and on-device optical cardiopulmonary vital sign measurement approach. It leverages a novel multi-task temporal shift convolutional attention network (MTTS-CAN) and enables real-time cardiovascular and respiratory measurements on mobile platforms. We evaluate our system on an Advanced RISC Machine (ARM) CPU and achieve state-of-the-art accuracy while running at over 150 frames per second which enables real-time applications. Systematic experimentation on large benchmark datasets reveals that our approach leads to substantial (20%-50%) reductions in error and generalizes well across datasets.

Input is a significant problem for wearable devices, particularly for head-mounted virtual and augmented reality systems. Contemporary AR/VR systems use in-air gestures or handheld controllers for interactivity. However, mid-air handwriting provides a natural, subtle, and easy-to-use way to input commands and text. In this demo, we propose and investigate ViFin, a new technique for input commands and text entry which tracks continuous micro finger-level writing with a commodity smartwatch through vibrations. Inspired by the recurrent neural aligner and transfer learning, ViFin recognizes continuous finger writing and works across different users and achieves an accuracy of 90% and 91% for recognizing numbers and letters, respectively. Finally, a real-time writing system with two specific applications using AR smartglasses are implemented.

The ability to track handheld controllers in 3D space is critical for interaction with head-mounted displays, such as those used in virtual and augmented reality systems. Today's systems commonly rely on dedicated infrastructure to track the controller or only provide inertial-based rotational tracking, which severely limits the user experience. Optical inside-out systems offer mobility but require line-of-sight and bulky tracking rings, which limit the ubiquity of these devices. In this work, we present Aura, an inside-out electromagnetic 6-DoF tracking system for handheld controllers. The tracking system consists of three coils embedded in a head-mounted display and a set of orthogonal receiver coils embedded in a handheld controller. We propose a novel closed-form and computationally simple tracking approach to reconstruct position and orientation in real time. Our handheld controller is small enough to fit in a pocket and consumes 45 mW of power, allowing it to operate for multiple days on a typical battery. An evaluation study demonstrates that Aura achieves a median tracking error of 5.5 mm and 0.8 degrees in 3D space within arm's reach.

Text messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not used a chatbot yet. Hence, understanding the usage patterns of first-time users can potentially inform and guide the design of future chatbots. In this paper, we report the findings of a study with 16 first-time chatbot users interacting with eight chatbots over multiple sessions on the Facebook Messenger platform. Analysis of chat logs and user interviews revealed that users preferred chatbots that provided either a 'human-like' natural language conversation ability, or an engaging experience that exploited the benefits of the familiar turn-based messaging interface. We conclude with implications to evolve the design of chatbots, such as: clarify chatbot capabilities, sustain conversation context, handle dialog failures, and end conversations gracefully.

Text messaging-based conversational systems, popularly called chatbots, have seen massive growth lately. Recent work on evaluating chatbots has found that there exists a mismatch between the chatbot's state of understanding (also called context) and the user's perception of the chatbot's understanding. Users found it difficult to use chatbots for complex tasks as the users were uncertain of the chatbots' intelligence level and contextual state. In this work, we propose Convey (CONtext View), a window added to the chatbot interface, displaying the conversational context and providing interactions with the context values. We conducted a usability evaluation of Convey with 16 participants. Participants preferred using chatbot with Convey and found it to be easier to use, less mentally demanding, faster, and more intuitive compared to a default chatbot without Convey. The paper concludes with a discussion of the design implications offered by Convey.

Breathalyzers, the standard quantitative method for assessing inebriation, are primarily owned by law enforcement and used only after a potentially inebriated individual is caught driving. However, not everyone has access to such specialized hardware. We present drunk user interfaces: smartphone user interfaces that measure how alcohol affects a person's motor coordination and cognition using performance metrics and sensor data. We examine five drunk user interfaces and combine them to form the "DUI app". DUI uses machine learning models trained on human performance metrics and sensor data to estimate a person's blood alcohol level (BAL). We evaluated DUI on 14 individuals in a week-long longitudinal study wherein each participant used DUI at various BALs. We found that with a global model that accounts for user-specific learning, DUI can estimate a person's BAL with an absolute mean error of 0.005% ± 0.007% and a Pearson's correlation coefficient of 0.96 with breathalyzer measurements.

Although cost-effective at-home blood pressure monitors are available, a complementary mobile solution can ease the burden of measuring BP at critical points throughout the day. In this work, we developed and evaluated a smartphone-based BP monitoring application called textitSeismo. The technique relies on measuring the time between the opening of the aortic valve and the pulse later reaching a periphery arterial site. It uses the smartphone's accelerometer to measure the vibration caused by the heart valve movements and the smartphone's camera to measure the pulse at the fingertip. The system was evaluated in a nine participant longitudinal BP perturbation study. Each participant participated in four sessions that involved stationary biking at multiple intensities. The Pearson correlation coefficient of the blood pressure estimation across participants is 0.20-0.77 ($mu$=0.55, $sigma$=0.19), with an RMSE of 3.3-9.2 mmHg ($mu$=5.2, $sigma$=2.0).

We present CASPER, a charging solution to enable a future of wearable devices that are much more distributed on the body. Instead of having to charge every device we want to adorn our bodies with, may it be distributed health sensors or digital jewelry, we can instead augment everyday objects such as beds, seats, and frequently worn clothing to provide convenient charging base stations that will charge devices on our body serendipitously as we go about our day. Our system works by treating the human body as a conductor and capacitively charging devices worn on the body whenever a well coupled electrical path is created during natural use of everyday objects. In this paper, we performed an extensive parameter characterization for through-body power transfer and based on our empirical findings, we present a design trade-off visualization to aid designers looking to integrate our system. Furthermore, we demonstrate how we utilized this design process in the development of our own smart bandage device and a LED adorned temporary tattoo that charges at hundreds of micro-watts using our system.

In 2017, smart speakers (such as Amazon Echo, Google Home, etc.) became a commercial success. Most smart speakers have a circular microphone array to provide hands-free, voice-only interaction from a distance. In this work, we exploit this mic array for opportunistically sensing gestures and tracking exercises. To this end, we measure the Doppler shift on a pilot tone caused by a gesturing human body, and use beamforming of the mic array to extend the range of the detection. Data from 12 participants show that gestures can be detected with an accuracy of 96.8% up to a distance of 2.5 meters using an inaudible 20 kHz pilot tone. For exercise tracking, we train a deep neural network to recognize 10 different exercises, and count repetitions by peak-finding heuristics. Data from 17 participants show that exercise classification accuracy is 96% and count accuracy is 91.8%. To conclude, we discuss hardware enhancements to smart speakers to further increase their gesture sensing capabilities.

Recent work has shown that fast, compact low-bitwidth neural networks can be surprisingly accurate. These networks use homogeneous binarization: all parameters in each layer or (more commonly) the whole model have the same low bitwidth (e.g., 2 bits). However, modern hardware allows efficient designs where each arithmetic instruction can have a custom bitwidth, motivating heterogeneous binarization, where every parameter in the network may have a different bitwidth. In this paper, we show that it is feasible and useful to select bitwidths at the parameter granularity during training. For instance a heterogeneously quantized version of modern networks such as AlexNet and MobileNet, with the right mix of 1-, 2- and 3-bit parameters that average to just 1.4 bits can equal the accuracy of homogeneous 2-bit versions of these networks. Further, we provide analyses to show that the heterogeneously binarized systems yield FPGA- and ASIC-based implementations that are correspondingly more efficient in both circuit area and energy efficiency than their homogeneous counterparts.

Sleep is an important aspect of our health, but it is difficult for people to track manually because it is an unconscious activity. The ability to sense sleep has aimed to lower the barriers of tracking sleep. Although sleep sensors are widely available, their usefulness and potential to promote healthy sleep behaviors has not been fully realized. To understand people's perspectives on sleep sensing devices and their potential for promoting sleep health, we surveyed 87 and interviewed 12 people who currently use or have previously used sleep sensors, interviewed 5 sleep medical experts, and conducted an in-depth qualitative analysis of 6986 reviews of the most popular commercial sleep sensing technologies. We found that the feedback provided by current sleep sensing technologies affects users' perceptions of their sleep and encourages goals that are in tension with evidence-based methods for promoting good sleep health. Our research provides design recommendations for improving the feedback of sleep sensing technologies by bridging the gap between expert and user goals.

Standard vehicle infotainment systems often include touch screens that allow the driver to control their mobile phone, navigation, audio, and vehicle configurations. For the driver's safety, these interfaces are often disabled or simplified while the car is in motion. Although this reduced functionality aids in reducing distraction for the driver, it also disrupts the usability of infotainment systems for passengers. Current infotainment systems are unaware of the seating position of their user and hence, cannot adapt. We present Carpacio, a system that takes advantage of the capacitive coupling created between the touchscreen and the electrode present in the seat when the user touches the capacitive screen. Using this capacitive coupling phenomenon, a car infotainment system can intelligently distinguish who is interacting with the screen seamlessly, and adjust its user interface accordingly. Manufacturers can easily incorporate Carpacio into vehicles since the included seat occupancy detection sensor or seat heating coils can be used as the seat electrode. We evaluated Carpacio in eight different cars and five mobile devices and found that it correctly detected over 2600 touches with an accuracy of 99.4%.

With the resurgence of head-mounted displays for virtual reality, users need new input devices that can accurately track their hands and fingers in motion. We introduce Finexus, a multipoint tracking system using magnetic field sensing. By instrumenting the fingertips with electromagnets, the system can track fine fingertip movements in real time using only four magnetic sensors. To keep the system robust to noise, we operate each electromagnet at a different frequency and leverage bandpass filters to distinguish signals attributed to individual sensing points. We develop a novel algorithm to efficiently calculate the 3D positions of multiple electromagnets from corresponding field strengths. In our evaluation, we report an average accuracy of 1.33 mm, as compared to results from an optical tracker. Our real-time implementation shows Finexus is applicable to a wide variety of human input tasks, such as writing in the air.

Cost and accessibility have impeded the adoption of spirometers (devices that measure lung function) outside clinical settings, especially in low-resource environments. Prior work, called SpiroSmart, used a smartphone's built-in microphone as a spirometer. However, individuals in low- or middle-income countries do not typically have access to the latest smartphones. In this paper, we investigate how spirometry can be performed from any phone-using the standard telephony voice channel to transmit the sound of the spirometry effort. We also investigate how using a 3D printed vortex whistle can affect the accuracy of common spirometry measures and mitigate usability challenges. Our system, coined SpiroCall, was evaluated with 50 participants against two gold standard medical spirometers. We conclude that SpiroCall has an acceptable mean error with or without a whistle for performing spirometry, and advantages of each are discussed.

We describe techniques that allow inexpensive, ultra-thin, battery-free Radio Frequency Identification (RFID) tags to be turned into simple paper input devices. We use sensing and signal processing techniques that determine how a tag is being manipulated by the user via an RFID reader and show how tags may be enhanced with a simple set of conductive traces that can be printed on paper, stencil-traced, or even hand-drawn. These traces modify the behavior of contiguous tags to serve as input devices. Our techniques provide the capability to use off-the-shelf RFID tags to sense touch, cover, overlap of tags by conductive or dielectric (insulating) materials, and tag movement trajectories. Paper prototypes can be made functional in seconds. Due to the rapid deployability and low cost of the tags used, we can create a new class of interactive paper devices that are drawn on demand for simple tasks. These capabilities allow new interactive possibilities for pop-up books and other papercraft objects.

Technologies that allow autonomous robots and computer systems to quickly recognize and interact with individuals in a group setting has the potential to enable a wide range of personalized experiences. However, existing solutions fail to both identify and locate individuals with enough speed to enable seamless interactions in very dynamic environments that require fast, implicit, non-intrusive, and ubiquitous recognition of users. In this work, we present a hybrid computer vision and RFID system that uses a novel reverse synthetic aperture technique to recover the relative motion paths of an RFID tags worn by people and correlate that to physical motion paths of individuals as measured with a 3D depth camera. Results show that our real-time system is capable of simultaneously recognizing and correctly assigning IDs to individuals within 4 seconds with 96.6% accuracy and groups of five people in 7 seconds with 95% accuracy. In order to test the effectiveness of this approach in realistic scenarios, groups of five participants play an interactive quiz game with an autonomous robot, resulting in an ID assignment accuracy of 93.3%.

While people often use smartphones to achieve specific goals, at other times they use them out of habit or to pass the time. Uses and Gratifications Theory explains that users' motivations for engaging with technology can be divided into instrumental and ritualistic purposes. Instrumental uses of technology are goal-directed and purposeful, while ritualistic uses are habitual and diversionary. In this paper, we provide an empirical account of the nature of instrumental vs. ritualistic use of smartphones based on data collected from 43 Android users over 2 weeks through logging application use and collecting ESM survey data about the purpose of use. We describe the phone-use behaviors users exhibit when seeking instrumental and ritualistic gratifications, and we develop a classification scheme for predicting ritualistic vs. instrumental use with an accuracy of 77% for a general model, increasing to more than 97% with a sliding confidence threshold. We discuss how such a model might be used to improve the experience of smartphone users in application areas such as recommender systems and social media.

We present HemaApp, a smartphone application that noninvasively monitors blood hemoglobin concentration using the smartphone's camera and various lighting sources. Hemoglobin measurement is a standard clinical tool commonly used for screening anemia and assessing a patient's response to iron supplement treatments. Given a light source shining through a patient's finger, we perform a chromatic analysis, analyzing the color of their blood to estimate hemoglobin level. We evaluate HemaApp on 31 patients ranging from 6 -- 77 years of age, yielding a 0.82 rank order correlation with the gold standard blood test. In screening for anemia, HemaApp achieve a sensitivity and precision of 85.7% and 76.5%. Both the regression and classification performance compares favorably with our control, an FDA-approved noninvasive hemoglobin measurement device. We also evaluate and discuss the effect of using different kinds of lighting sources.

Serious brain injuries, spinal injuries, and motor neuron diseases often lead to severe paralysis. Individuals with such disabilities can benefit from interaction techniques that enable them to interact with the devices and thereby the world around them. While a number of systems have proposed tongue-based gesture detection systems, most of these systems require intrusive instrumentation of the user's body (e.g., tongue piercing, dental retainers, multiple electrodes on chin). In this paper, we propose a wireless, non-intrusive and non-contact facial gesture detection system using X-band Doppler. The system can accurately differentiate between 8 different facial gestures through non-contact sensing, with an average accuracy of 94.3%.

Many HCI and ubiquitous computing systems are characterized by two important properties: their output is uncertain-it has an associated accuracy that researchers attempt to optimize-and this uncertainty is user-facing-it directly affects the quality of the user experience. Novel classifiers are typically evaluated using measures like the F1 score-but given an F-score of (e.g.) 0.85, how do we know whether this performance is good enough? Is this level of uncertainty actually tolerable to users of the intended application-and do people weight precision and recall equally? We set out to develop a survey instrument that can systematically answer such questions. We introduce a new measure, acceptability of accuracy, and show how to predict it based on measures of classifier accuracy. Out tool allows us to systematically select an objective function to optimize during classifier evaluation, but can also offer new insights into how to design feedback for user-facing classification systems (e.g., by combining a seemingly-low-performing classifier with appropriate feedback to make a highly usable system). It also reveals potential issues with the ubiquitous F1-measure as applied to user-facing systems.

Smartphones and tablets are often used in dynamic environments that force users to break focus and attend to their surroundings, creating a form of "situational impairment." Current mobile devices have no ability to sense when users divert or restore their attention, let alone provide support for resuming tasks. We therefore introduce SwitchBack, a system that allows mobile device users to resume tasks more efficiently. SwitchBack is built upon Focus and Saccade Tracking (FAST), which uses the front-facing camera to determine when the user is looking and how their eyes are moving across the screen. In a controlled study, we found that FAST can identify how many lines the user has read in a body of text within a mean absolute percent error of just 3.9%. We then tested SwitchBack in a dual focus-of-attention task, finding that SwitchBack improved average reading speed by 7.7% in the presence of distractions.

Time of use tiered pricing schedules encourage shifting electricity demand from peak to off-peak hours. Charging times for electric vehicles (EV) can be shifted into overnight hours, which are usually off-peak. EVs can also be used as energy storage devices, available during certain peak hours to power a house with electricity stored during off-peak hours. Studies suggest both techniques are practical, but were based on simulated demand patterns or large commercial fleets. To investigate feasibility on a per home basis, we collected data from 15 EV homes using the Lab of Things sensing infrastructure. We evaluate a scheme that powers homes with their car battery during expensive electricity periods and then charges the battery during cheaper periods. We show an average potential savings of $10.91/month for shifting charging times, and an additional $13.58/month for powering the home from the EV, even accounting for the inefficiencies of electric conversion.

Emerging uses of imaging technology for consumers cover a wide range of application areas from health to interaction techniques; however, typical cameras primarily transduce light from the visible spectrum into only three overlapping components of the spectrum: red, blue, and green. In contrast, hyperspectral imaging breaks down the electromagnetic spectrum into more narrow components and expands coverage beyond the visible spectrum. While hyperspectral imaging has proven useful as an industrial technology, its use as a sensing approach has been fragmented and largely neglected by the UbiComp community. We explore an approach to make hyperspectral imaging easier and bring it closer to the end-users. HyperCam provides a low-cost implementation of a multispectral camera and a software approach that automatically analyzes the scene and provides a user with an optimal set of images that try to capture the salient information of the scene. We present a number of use-cases that demonstrate HyperCam's usefulness and effectiveness.

In this paper, we present DoppleSleep -- a contactless sleep sensing system that continuously and unobtrusively tracks sleep quality using commercial off-the-shelf radar modules. DoppleSleep provides a single sensor solution to track sleep-related physical and physiological variables including coarse body movements and subtle and fine-grained chest, heart movements due to breathing and heartbeat. By integrating vital signals and body movement sensing, DoppleSleep achieves 89.6% recall with Sleep vs. Wake classification and 80.2% recall with REM vs. Non-REM classification compared to EEG-based sleep sensing. Lastly, it provides several objective sleep quality measurements including sleep onset latency, number of awakenings, and sleep efficiency. The contactless nature of DoppleSleep obviates the need to instrument the user's body with sensors. Lastly, DoppleSleep is implemented on an ARM microcontroller and a smartphone application that are benchmarked in terms of power and resource usage.

The different electronic devices we use on a daily basis produce distinct electromagnetic radiation due to differences in their underlying electrical components. We present MagnifiSense, a low-power wearable system that uses three passive magneto-inductive sensors and a minimal ADC setup to identify the device a person is operating. MagnifiSense achieves this by analyzing near-field electromagnetic radiation from common components such as the motors, rectifiers, and modulators. We conducted a staged, in-the-wild evaluation where an instrumented participant used a set of devices in a variety of settings in the home such as cooking and outdoors such as commuting in a vehicle. MagnifiSense achieves a classification accuracy of 82.6% using a model-agnostic classifier and 94.0% using a model-specific classifier. In a 24-hour naturalistic deployment, MagnifiSense correctly identified 25 of the total 29 events, while achieving a low false positive rate of 0.65% during 20.5 hours of non-activity.

Electricity and appliance usage information can often reveal the nature of human activities in a home. For instance, sensing the use of vacuum cleaner, a microwave oven, and kitchen appliances can give insights into a person's current activities. Instead of putting a sensor on each appliance, our technique is based on the idea that appliance usage can be sensed by their manifestations in an environment's existing electrical infrastructure. Prior approaches using this technique could only detect an appliance's on-off states; that is, they only sense “what” is being used, but not “how” it is used. In this paper, we introduce DOSE, a significant advancement for inferring operating states of electronic devices from a single sensing point in a home. When an electronic device is in operation, it generates time-varying Electromagnetic Interference (EMI) based upon its operating states (e.g., vacuuming on a rug vs. hardwood floor). This EMI noise is coupled to the power line and can be picked up from a single sensing hardware attached to the wall outlet in a house. Unlike prior data-driven approaches, we employ domain knowledge of the device's circuitry for semi-supervised model training to avoid tedious labeling process. We evaluated DOSE in a residential house for 2 months and found that operating states for 16 appliances could be estimated with an average accuracy of 93.8%. These fine-grained electrical characteristics affords rich feature sets of electrical events and have the potential to support various applications such as in-home activity inference, energy disaggregation and device failure detection.

Sensing respiration rate has many applications in monitoring various health conditions, such as sleep apnea and chronic obstructive pulmonary disease. In this paper, we present WiBreathe, a wireless, high fidelity and non-invasive breathing monitor that leverages wireless signals at 2.4 GHz to estimate an individual's respiration rate. Our work extends past approaches of using wireless signals for respiratory monitoring by using only a single transmitter-receiver pair at the same frequency range of commodity Wi-Fi signals to estimate the respiratory rate of an individual. This is done irrespective of whether they are in line of sight or not (e.g., through walls). Furthermore, we demonstrate the capability of WiBreathe in detecting multiple people and by extension, their respiration rates. We evaluate our approach in various natural environments and show that we can track breathing with the accuracy of 1.54 breaths per minute when compared to a clinical respiratory chest band.

We present SurfaceLink, a system where users can make natural surface gestures to control association and information transfer among a set of devices that are placed on a mutually shared surface (e.g., a table). SurfaceLink uses a combination of on-device accelerometers, vibration motors, speakers and microphones (and, optionally, an off-device contact microphone for greater sensitivity) to sense gestures performed on the shared surface. In a controlled evaluation with 10 participants, SurfaceLink detected the presence of devices on the same surface with 97.7% accuracy, their relative arrangement with 89.4% accuracy, and various single- and multi-touch surface gestures with an average accuracy of 90.3%. A usability analysis showed that SurfaceLink has advantages over current multi-device interaction techniques in a number of situations.

In this paper, we present a significant improvement over past work on non-contact end-user deployable sensor for real time whole home power consumption. The technique allows users to place a single device consisting of magnetic pickups on the outside of a power or breaker panel to infer whole home power consumption without the need for professional installation of current transformers (CTs). The new approach does not require precise placement on the breaker panel, a key requirement in previous approaches. This is enabled through a self-calibration technique using a neural network that dynamically learns the transfer function despite the placement of the sensor and the construction of the breaker panel itself. We also demonstrate the ability to actually infer true power using this technique, unlike past solutions that have only been able to capture apparent power. We have evaluated our technique in six homes and one industrial building, including one seven-day deployment. Our results show we can estimate true power consumption with an average accuracy of 95.0% during naturalistic energy use in the home.

We introduce AirLink, a novel technique for sharing files between multiple devices. By waving a hand from one device towards another, users can directly transfer files between them. The system utilizes the devices' built-in speakers and microphones to enable easy file sharing between phones, tablets and laptops. We evaluate our system in an 11-participant study with 96.8% accuracy, showing the feasibility of using AirLink in a multiple-device environment. We also implemented a real-time system and demonstrate the capability of AirLink in various applications.

Health sensing through smartphones has received considerable attention in recent years because of the devices' ubiquity and promise to lower the barrier for tracking medical conditions. In this paper, we focus on using smartphones to monitor newborn jaundice, which manifests as a yellow discoloration of the skin. Although a degree of jaundice is common in healthy newborns, early detection of extreme jaundice is essential to prevent permanent brain damage or death. Current detection techniques, however, require clinical tests with blood samples or other specialized equipment. Consequently, newborns often depend on visual assessments of their skin color at home, which is known to be unreliable. To this end, we present BiliCam, a low-cost system that uses smartphone cameras to assess newborn jaundice. We evaluated BiliCam on 100 newborns, yielding a 0.85 rank order correlation with the gold standard blood test. We also discuss usability challenges and design solutions to make the system practical.

Power remains a challenge in the widespread deployment of long-lived wireless sensing systems, which has led researchers to consider power harvesting as a potential solution. In this paper, we present a thermal power harvester that utilizes naturally changing ambient temperature in the environment as the power source. In contrast to traditional thermoelectric power harvesters, our approach does not require a spatial temperature gradient; instead it relies on temperature fluctuations over time, enabling it to be used freestanding in any environment in which temperature changes throughout the day. By mechanically coupling linear motion harvesters with a temperature sensitive bellows, we show the capability of harvesting up to 21 mJ of energy per cycle of temperature variation within the range 5 °C to 25 °C. We also demonstrate the ability to power a sensor node, transmit sensor data wirelessly, and update a bistable E-ink display after as little as a 0.25 °C ambient temperature change.

Current smartphone inputs are limited to physical buttons, touchscreens, cameras or built-in sensors. These approaches either require a dedicated surface or line-of-sight for interaction. We introduce SideSwipe, a novel system that enables in-air gestures both above and around a mobile device. Our system leverages the actual GSM signal to detect hand gestures around the device. We developed an algorithm to convert the discrete and bursty GSM pulses to a continuous wave that can be used for gesture recognition. Specifically, when a user waves their hand near the phone, the hand movement disturbs the signal propagation between the phone's transmitter and added receiving antennas. Our system captures this variation and uses it for gesture recognition. To evaluate our system, we conduct a study with 10 participants and present robust gesture recognition with an average accuracy of 87.2% across 14 hand gestures.

Current solutions for enabling touch interaction on existing non-touch LCD screens require adding additional sensors to the interaction surface. We present uTouch, a system that detects and classifies touches and hovers without any modification to the display, and without adding any sensors to the user. Our approach utilizes existing signals in an LCD that are amplified when a user brings their hand near or touches the LCD's front panel. These signals are coupled onto the power lines, where they appear as electromagnetic interference (EMI) which can be sensed using a single device connected elsewhere on the power line infrastructure. We validate our approach with an 11 user, 8 LCD study, and demonstrate a real-time system.

The challenge of mobile text entry is exacerbated as mobile devices are used in a number of situations and with a number of hand postures. We introduce ContextType, an adaptive text entry system that leverages information about a user's hand posture (using two thumbs, the left thumb, the right thumb, or the index finger) to improve mobile touch screen text entry. ContextType switches between various keyboard models based on hand posture inference while typing. ContextType combines the user's posture-specific touch pattern information with a language model to classify the user's touch events as pressed keys. To create our models, we collected usage patterns from 16 participants in each of the four postures. In a subsequent study with the same 16 participants comparing ContextType to a control condition, ContextType reduced total text entry error rate by 20.6%.

Mobile and embedded electronics are pervasive in today's environment. As such, it is necessary to have a natural and intuitive way for users to indicate the intent to connect to these devices from a distance. We present DopLink, an ultrasonic-based device selection approach. It utilizes the already embedded audio hardware in smart devices to determine if a particular device is being pointed at by another device (i.e., the user waves their mobile phone at a target in a pointing motion). We evaluate the accuracy of DopLink in a controlled user study, showing that, within 3 meters, it has an average accuracy of 95% for device selection and 97% for finding relative device position. Finally, we show three applications of DopLink: rapid device pairing, home automation, and multi-display synchronization.

Input modalities such as speech and gesture allow users to interact with computers without holding or touching a physical device, thus enabling at-a-distance interaction. It remains an open problem, however, to incorporate haptic feedback into such interaction. In this work, we explore the use of air vortex rings for this purpose. Unlike standard jets of air, which are turbulent and dissipate quickly, vortex rings can be focused to travel several meters and impart perceptible feedback. In this paper, we review vortex formation theory and explore specific design parameters that allow us to generate vortices capable of imparting haptic feedback. Applying this theory, we developed a prototype system called AirWave. We show through objective meas urements that AirWave can achieve spatial resolution of less than 10 cm at a distance of 2.5 meters. We further demonstrate through a user study that this can be used to direct tactile stimuli to different regions of the human body.

This paper presents WiSee, a novel gesture recognition system that leverages wireless signals (e.g., Wi-Fi) to enable whole-home sensing and recognition of human gestures. Since wireless signals do not require line-of-sight and can traverse through walls, WiSee can enable whole-home gesture recognition using few wireless sources. Further, it achieves this goal without requiring instrumentation of the human body with sensing devices. We implement a proof-of-concept prototype of WiSee using USRP-N210s and evaluate it in both an office environment and a two- bedroom apartment. Our results show that WiSee can identify and classify a set of nine gestures with an average accuracy of 94%.

While much progress has been made in wearable computing in recent years, input techniques remain a key challenge. In this paper, we introduce uTrack, a technique to convert the thumb and fingers into a 3D input system using magnetic field (MF) sensing. A user wears a pair of magnetometers on the back of their fingers and a permanent magnet affixed to the back of the thumb. By moving the thumb across the fingers, we obtain a continuous input stream that can be used for 3D pointing. Specifically, our novel algorithm calculates the magnet's 3D position and tilt angle directly from the sensor readings. We evaluated uTrack as an input device, showing an average tracking accuracy of 4.84 mm in 3D space - sufficient for subtle interaction. We also demonstrate a real-time prototype and example applications allowing users to interact with the computer using 3D finger input.

Vibrotactile devices suffer from poor energy efficiency, arising from a mismatch between the device and the impedance of the human skin. This results in over-sized actuators and excessive power consumption, and prevents development of more sophisticated, miniaturized and low-power mobile tactile devices. In this paper, we present the experimental evaluation of a vibrotactile system designed to match the impedance of the skin to the impedance of the actuator. This system is able to quadruple the motion of the skin without increasing power consumption, and produce sensations equivalent to a standard system while consuming 1/2 of the power. By greatly reducing the size and power constraints of vibrotactile actuators, this technology offers a means to realize more sophisticated, smaller haptic devices for the user interface community.

Haptic devices that provide robust and realistic force feedback are generally grounded to counterweight the applied force, prohibiting their use in mobile devices. Many ungrounded force-feedback devices rely on the gyro effect to produce torques on the human body, but their active control systems render them extremely bulky for implementation in small mobile devices. We present GyroTab, a relatively flat handheld system that utilizes the gyro effect to provide torque feedback. GyroTab relies on the user to produce an input torque and provides feedback by opposing that torque, making its feedback reactive to the user's motion. We describe the implementation of GyroTab, discuss the kinds of feedback it generates, and explore some of the psychophysical results we obtained from a study with the device.

Computer vision and inertial measurement have made it possible for people to interact with computers using whole-body gestures. Although there has been rapid growth in the uses and applications of these systems, their ubiquity has been limited by the high cost of heavily instrumenting either the environment or the user. In this paper, we use the human body as an antenna for sensing whole-body gestures. Such an approach requires no instrumentation to the environment, and only minimal instrumentation to the user, and thus enables truly mobile applications. We show robust gesture recognition with an average accuracy of 93% across 12 whole-body gestures, and promising results for robust location classification within a building. In addition, we demonstrate a real-time interactive system which allows a user to interact with a computer using whole-body gestures

Few means currently exist for home occupants to learn about their water consumption: e.g., where water use occurs, whether such use is excessive and what steps can be taken to conserve. Emerging water sensing systems, however, can provide detailed usage data at the level of individual water fixtures (i.e., disaggregated usage data). In this paper, we perform formative evaluations of two sets of novel eco-feedback displays that take advantage of this disaggregated data. The first display set isolates and examines specific elements of an eco-feedback design space such as data and time granularity. Displays in the second set act as design probes to elicit reactions about competition, privacy, and integration into domestic space. The displays were evaluated via an online survey of 651 North American respondents and in-home, semi-structured interviews with 10 families (20 adults). Our findings are relevant not only to the design of future water eco-feedback systems but also for other types of consumption (e.g., electricity and gas).

Gesture is becoming an increasingly popular means of interacting with computers. However, it is still relatively costly to deploy robust gesture recognition sensors in existing mobile platforms. We present SoundWave, a technique that leverages the speaker and microphone already embedded in most commodity devices to sense in-air gestures around the device. To do this, we generate an inaudible tone, which gets frequency-shifted when it reflects off moving objects like the hand. We measure this shift with the microphone to infer various gestures. In this note, we describe the phenomena and detection algorithm, demonstrate a variety of gestures, and present an informal evaluation on the robustness of this approach across different devices and people.

In-home sensing and inference systems impose privacy risks and social tensions, which can be substantial barriers for the wide adoption of these systems. To understand what might affect people's perceptions and acceptance of in-home sensing and inference systems, we conducted an empirical study with 22 participants from 11 households. The study included in-lab activities, four weeks using sensor proxies in situ, and exit interviews. We report on participants' perceived benefits and concerns of in-home sensing applications and the observed changes of their perceptions throughout the study. We also report on tensions amongst stakeholders around the adoption and use of such systems. We conclude with a discussion on how the ubicomp design space might be sensitized to people's perceived concerns and tensions regarding sensing and inference in the home.

Wearable sensor systems have been used in the ubiquitous computing community and elsewhere for applications such as activity and gesture recognition, health and wellness monitoring, and elder care. Although the power consumption of accelerometers has already been highly optimized, this work introduces a novel sensing approach which lowers the power requirement for motion sensing by orders of magnitude. We present an ultra-low-power method for passively sensing body motion using static electric fields by measuring the voltage at any single location on the body. We present the feasibility of using this sensing approach to infer the amount and type of body motion anywhere on the body and demonstrate an ultra-low-power motion detector used to wake up more power-hungry sensors. The sensing hardware consumes only 3.3 μW, and wake-up detection is done using an additional 3.3 μW (6.6 μW total).

Home spirometry is gaining acceptance in the medical community because of its ability to detect pulmonary exacerbations and improve outcomes of chronic lung ailments. However, cost and usability are significant barriers to its widespread adoption. To this end, we present SpiroSmart, a low-cost mobile phone application that performs spirometry sensing using the built-in microphone. We evaluate SpiroSmart on 52 subjects, showing that the mean error when compared to a clinical spirometer is 5.1% for common measures of lung function. Finally, we show that pulmonologists can use SpiroSmart to diagnose varying degrees of obstructive lung ailments.

We introduce GripSense, a system that leverages mobile device touchscreens and their built-in inertial sensors and vibration motor to infer hand postures including one- or two-handed interaction, use of thumb or index finger, or use on a table. GripSense also senses the amount of pres-sure a user exerts on the touchscreen despite a lack of direct pressure sensors by inferring from gyroscope readings when the vibration motor is "pulsed." In a controlled study with 10 participants, GripSense accurately differentiated device usage on a table vs. in hand with 99.67% accuracy and when in hand, it inferred hand postures with 84.26% accuracy. In addition, GripSense distinguished three levels of pressure with 95.1% accuracy. A usability analysis of GripSense was conducted in three custom applications and showed that pressure input and hand-posture sensing can be useful in a number of scenarios.

We conduct an extensive study of information leakage over the powerline infrastructure from eight televisions (TVs) spanning multiple makes, models, and underlying technologies. In addition to being of scientific interest, our findings contribute to the overall debate of whether or not measurements of residential powerlines reveal significant information about the activities within a home. We find that the power supplies of modern TVs produce discernible electromagnetic interference (EMI) signatures that are indicative of the video content being displayed. We measure the stability of these signatures over time and across multiple instances of the same TV model, as well as the robustness of these signatures in the presence of other noisy electronic devices connected to the same powerline.

We present Interactive Generator (InGen), a self-powered wireless rotary input device capable of generating haptic or force feedback without the need for any external power source. Our approach uses a modified servomotor to perform three functions: (1) generating power for wireless communication and embedded electronics, (2) sensing the direction and speed of rotation, and (3) providing force feedback during rotation. While InGen is rotating, the device is capable of providing the sensation of detents or bumps, changes in stiffness, and abrupt stops using only power that is harvested during interaction. We describe the device in detail, demonstrate an initial 'TV remote control' application, and end with a discussion of our experiences developing the prototype and application. To the best of our knowledge, InGen is the first self-powered device, which also provides haptic feedback during operation. More broadly, this work demonstrates a new class of input sys-tems that uses human-generated power to provide feedback to the user and wirelessly communicate sensed information.

Touch sensing and computer vision have made human-computer interaction possible in environments where keyboards, mice, or other handheld implements are not available or desirable. However, the high cost of instrumenting environments limits the ubiquity of these technologies, particularly in home scenarios where cost constraints dominate installation decisions. Fortunately, home environments frequently offer a signal that is unique to locations and objects within the home: electromagnetic noise. In this work, we use the body as a receiving antenna and leverage this noise for gestural interaction. We demonstrate that it is possible to robustly recognize touched locations on an uninstrumented home wall using no specialized sensors. We conduct a series of experiments to explore the capabilities that this new sensing modality may offer. Specifically, we show robust classification of gestures such as the position of discrete touches around light switches, the particular light switch being touched, which appliances are touched, differentiation between hands, as well as continuous proximity of hand to the switch, among others. We close by discussing opportunities, limitations, and future work.

We present the Haptic Laser, a system for providing a range of tactile sensations to represent a physical environment at-a-distance. The Haptic Laser is a handheld device that simulates interaction with physical surfaces as a user targets objects of interest (e.g., a light switch, TV, etc). Using simple computer vision techniques for scene analysis and laser range finding for calculating distance, the Haptic Laser extracts information about the physical environment and conveys it haptically through a collection of hardware actuators. Pointing the Haptic Laser around a room, for example, presents the user with information about the presence of objects, transitions, and edges through touch rather than, or in addition to, vision. The Haptic Laser extends current work on haptic touch screens and pens, and is designed to allow for haptic feedback from a distance using multiple feedback channels.

We present HeatWave, a system that uses digital thermal imaging cameras to detect, track, and support user interaction on arbitrary surfaces. Thermal sensing has had limited examination in the HCI research community and is generally under-explored outside of law enforcement and energy auditing applications. We examine the role of thermal imaging as a new sensing solution for enhancing user surface interaction. In particular, we demonstrate how thermal imaging in combination with existing computer vision techniques can make segmentation and detection of routine interaction techniques possible in real-time, and can be used to complement or simplify algorithms for traditional RGB and depth cameras. Example interactions include (1) distinguishing hovering above a surface from touch events, (2) shape-based gestures similar to ink strokes, (3) pressure based gestures, and (4) multi-finger gestures. We close by discussing the practicality of thermal sensing for naturalistic user interaction and opportunities for future work.

In this paper, we describe LightWave, a sensing approach that turns ordinary compact fluorescent light (CFL) bulbs into sensors of human proximity. Unmodified CFL bulbs are shown to be sensitive proximity transducers when they are illuminated. This approach utilizes predictable variations in electromagnetic noise resulting from the change in impedance due to the proximity of a human body to the bulb. The electromagnetic noise can be sensed from any point along a home's electrical wiring. This allows users to perform gestures near any CFL lighting fixture, even when multiple lamps are operational. Gestures can be sensed using a single interface device plugged into any electrical outlet. We experimentally show that we can reliably detect hover gestures (waving a hand close to a lamp), touches on lampshades, and touches on the glass part of the bulb itself. Additionally, we show that touches anywhere along the body of a metal lamp can be detected. These basic detectable signals can then be combined to form complex gesture sequences for a variety of applications. We also show that CFLs can function as more general-purpose sensors for distributed human motion detection and ambient temperature sensing.

Audio-based cough detection has become more pervasive in recent years because of its utility in evaluating treatments and the potential to impact the quality of life for individuals with chronic cough. We critically examine the current state of the art in cough detection, concluding that existing approaches expose private audio recordings of users and bystanders. We present a novel algorithm for detecting coughs from the audio stream of a mobile phone. Our system allows cough sounds to be reconstructed from the feature set, but prevents speech from being reconstructed intelligibly. We evaluate our algorithm on data collected in the wild and report an average true positive rate of 92% and false positive rate of 0.5%. We also present the results of two psychoacoustic experiments which characterize the tradeoff between the fidelity of reconstructed cough sounds and the intelligibility of reconstructed speech.

We present an empirical study of the long-term practicality of using human motion to generate operating power for body-mounted consumer electronics and health sensors. We have collected a large continuous acceleration data set from eight experimental subjects going about their normal daily routine for three days each. Each subject is instrumented with a data collection apparatus that simultaneously logs 3-axis, 80 Hz acceleration data from six body locations. We use this data set to optimize a first-principles physical model of the commonly used velocity damped resonant generator (VDRG) by selecting physical parameters such as resonant frequency and damping coefficient to maximize the harvested power. Our results show that with reasonable assumptions on size, mass, placement, and efficiency of VDRG harvesters, most body-mounted wireless sensors and even some consumer electronics devices can be powered continuously and indefinitely from everyday motion. We have optimized the power harvesters for each individual and for each body location. In addition, we present the potential of designing a damping- and frequency-tunable power harvester that could mitigate the power reduction of a generator generalized for "average” subjects. We present the full details on the collection of the acceleration data sets, the development of the VDRG model, and a numerical simulator, and discuss some of the future challenges that remain in this promising field of research.

We present the design, development, and evaluation of an end-user installable, whole house power consumption sensing system capable of gathering accurate real-time power use that does not require installing a current transformer around the electrical feeds in a home. Rather, our sensor system offers contactless operation by simply placing it on the outside of the breaker panel in a home. Although there are a number of existing commercial systems for gathering energy use in a home, almost none can easily and safely be installed by a homeowner (especially for homes in the U.S.). Our approach leverages advances in magnetoresistive materials and circuit design to allow contactless operation by reliably sensing the magnetic field induced by the 60 Hz current and a closed loop circuit allows us to precisely infer the power consumption in real-time. The contribution of this work is an enabling technology for researchers in the fields of Ubiquitous Computing and Human-Computer Interaction wanting to conduct practical large-scale deployments of end-user-deployable energy monitoring applications. We discuss the technical details, the iterative design, and end-user evaluations of our sensing approach.

We present WATTR, a novel self-powered water activity sensor that utilizes residential water pressure impulses as both a powering and sensing source. Consisting of a power harvesting circuit, piezoelectric sensor, ultra-low-power 16-bit microcontroller, 16-bit analog-to-digital converter (ADC), and a 433 MHz wireless transmitter, WATTR is capable of sampling home water pressure at 33 Hz and transmitting over 3 m when any water fixture in the home is opened or closed. WATTR provides an alternative sensing solution to the power intensive Bluetooth-based sensor used in the HydroSense project by Froehlich et al. [2] for single-point whole-home water usage. We demonstrate WATTR as a viable self-powered sensor capable of monitoring and transmitting water usage data without the use of a battery. Unlike other water-based power harvesters, WATTR does not waste water to power itself. We discuss the design, implementation, and experimental verification of the WATTR device.

A persistent concern of wireless sensors is the power consumption required for communication, which presents a significant adoption hurdle for practical ubiquitous computing applications. This work explores the use of the home powerline as a large distributed antenna capable of receiving signals from ultra-low-power wireless sensor nodes and thus allowing nodes to be detected at ranges that are otherwise impractical with traditional over-the-air reception. We present the design and implementation of small ultra-low-power 27 MHz sensor nodes that transmit their data by coupling over the powerline to a single receiver attached to the powerline in the home. We demonstrate the ability of our general purpose wireless sensor nodes to provide whole-home coverage while consuming less than 1 mW of power when transmitting (65 ¼W consumed in our custom CMOS transmitter). This is the lowest power transmitter to date compared to those found in traditional whole-home wireless systems.

This paper presents ElectriSense, a new solution for automatically detecting and classifying the use of electronic devices in a home from a single point of sensing. ElectriSense relies on the fact that most modern consumer electronics and fluorescent lighting employ switch mode power supplies (SMPS) to achieve high efficiency. These power supplies continuously generate high frequency electromagnetic interference (EMI) during operation that propagates throughout a home's power wiring. We show both analytically and by in-home experimentation that EMI signals are stable and predictable based on the device's switching frequency characteristics. Unlike past transient noise-based solutions, this new approach provides the ability for EMI signatures to be applicable across homes while still being able to differentiate between similar devices in a home. We have evaluated our solution in seven homes, including one six-month deployment. Our results show that ElectriSense can identify and classify the usage of individual devices with a mean accuracy of 93.82%.

Modern automobiles are no longer mere mechanical devices; they are pervasively monitored and controlled by dozens of digital computers coordinated via internal vehicular networks. While this transformation has driven major advancements in efficiency and safety, it has also introduced a range of new potential risks. In this paper we experimentally evaluate these issues on a modern automobile and demonstrate the fragility of the underlying system structure. We demonstrate that an attacker who is able to infiltrate virtually any Electronic Control Unit (ECU) can leverage this ability to completely circumvent a broad array of safety-critical systems. Over a range of experiments, both in the lab and in road tests, we demonstrate the ability to adversarially control a wide range of automotive functions and completely ignore driver input\dash including disabling the brakes, selectively braking individual wheels on demand, stopping the engine, and so on. We find that it is possible to bypass rudimentary network security protections within the car, such as maliciously bridging between our car's two internal subnets. We also present composite attacks that leverage individual weaknesses, including an attack that embeds malicious code in a car's telematics unit and that will completely erase any evidence of its presence after a crash. Looking forward, we discuss the complex challenges in addressing these vulnerabilities while considering the existing automotive ecosystem.

Haptic feedback provides an additional interaction channel when auditory and visual feedback may not be appropriate. We present a novel haptic feedback system that changes its elasticity to convey information for eyes-free interaction. SqueezeBlock is an electro-mechanical system that can realize a virtual spring having a programmatically controlled spring constant. It also allows for additional haptic modalities by altering the Hooke's Law linear-elastic force- displacement equation, such as non-linear springs, size changes, and spring length (range of motion) variations. This ability to program arbitrarily spring constants also allows for "click" and button-like feedback. We present several potential applications along with results from a study showing how well participants can distinguish between several levels of stiffness, size, and range of motion. We conclude with implications for interaction design.

Despite increased knowledge about how Information and Communications Technologies (ICTs) are used to support religious and spiritual practices, designers know little about how to design technologies for faith-related purposes. Our research suggests incorporating sacred imagery into techno-spiritual applications can be useful in guiding development. We illustrate this through the design and evaluation of a mobile phone application developed to support Islamic prayer practices. Our contribution is to show how religious imagery can be used in the design of applications that go beyond the provision of functionality to connect people to the experience of religion.

Recent work has examined infrastructure-mediated sensing as a practical, low-cost, and unobtrusive approach to sensing human activity in the physical world. This approach is based on the idea that human activities (e.g., running a dishwasher, turning on a reading light, or walking through a doorway) can be sensed by their manifestations in an environment's existing infrastructures (e.g., a home's water, electrical, and HVAC infrastructures). This paper presents HydroSense, a low-cost and easily-installed single-point sensor of pressure within a home's water infrastructure. HydroSense supports both identification of activity at individual water fixtures within a home (e.g., a particular toilet, a kitchen sink, a particular shower) as well as estimation of the amount of water being used at each fixture. We evaluate our approach using data collected in ten homes. Our algorithms successfully identify fixture events with 97.9% aggregate accuracy and can estimate water usage with error rates that are comparable to empirical studies of traditional utility-supplied water meters. Our results both validate our approach and provide a basis for future improvements.

Within a group of peers, it is often useful or interesting to know whether someone in the group has gone to bed or whether they have awakened in the morning. This information, naturally integrated as a peripheral augmentation of an alarm clock, allows people to know whether it is appropriate to make a call or feel more connected with someone living remotely. In this paper, we present the design and evaluation of such an alarm clock, the BuddyClock, and describe how it enables users in a small social network to automatically share information about their sleeping behaviors with one another. Through 3-6 week deployment studies of this technology with five different social networks, we found that the alarm clock affected participant behaviors and allowed them to feel more connected to those with whom they shared their sleeping behaviors.

Fingerprinting techniques for indoor localization have been widely explored. A particular approach by Patel et al. suggested leveraging of the residential powerline as the signaling mechanism for a domestic location capability. In this paper, we critically examine that initial work, called powerline positioning (PLP). We find the proposed technique lacking in temporal stability, requiring frequent and undesired recalibration in some environments. We also determine that there is no a priori method to determine a pair of signaling frequencies that will reliably work in any space. We propose a wideband approach to PLP (WPLP) that injects up to 44 different frequencies into the powerline. We show that this WPLP approach improves upon overall positioning accuracy, demonstrates greatly improved temporal stability and has the added advantage of working in commercial indoor spaces.

We present an empirical study of the long-term practicality of using human motion to generate operating power for body-mounted consumer electronics and health sensors. We have collected a large continuous acceleration dataset from eight experimental subjects going about their normal daily routine for 3 days each. Each subject is instrumented with a data collection apparatus that simultaneously logs 3-axis, 80Hz acceleration data from six body locations. We use this dataset to optimize a first-principles physical model of the commonly used velocity damped resonant generator (VDRG) by selecting physical parameters such as resonant frequency and damping coefficient to maximize harvested power. Our results show that with reasonable assumptions on size, mass, placement, and efficiency of VDRG harvesters, most body-mounted wireless sensors and even some consumer electronics devices, may be powered continuously and indefinitely from everyday motion.

From birth through age five, children undergo rapid development and learn skills that will influence them their entire lives. Regular visits to the pediatrician and detailed record-keeping can ensure that children are progressing and can identify early warning signs of developmental delay or disability. However, new parents are often overwhelmed with new responsibilities, and we believe there is an opportunity for computing technology to assist in this process. In this paper, we present a qualitative study aimed at uncovering some specific needs for record-keeping and analysis for new parents and their network of caregivers. Through interviews and focus groups, we have confirmed assumptions about the rationales parents have and the functions required for using technology for record-keeping. We also identify new themes, potential prototypes, and design guidelines for this domain.

We describe a unique form of hands-free interaction that can be implemented on most commodity computing platforms. Our approach supports blowing at a laptop or computer screen to directly control certain interactive applications. Localization estimates are produced in real-time to determine where on the screen the person is blowing. Our approach relies solely on a single microphone, such as those already embedded in a standard laptop or one placed near a computer monitor, which makes our approach very cost-effective and easy-to-deploy. We show example interaction techniques that leverage this approach.

Finding lost items is a common problem for the visually impaired and is something that computing technology can help alleviate. In this paper, we present the design and evaluation of a mobile solution, called FETCH, for allowing the visually impaired to track and locate objects they lose frequently but for which they do not have a specific strategy for tracking. FETCH uses devices the user already owns, such as their cell phone or laptop, to locate objects around their house. Results from a focus group with visually impaired users informed the design of the system. We then studied the usability of a laptop solution in a laboratory study and studied the usability and usefulness of the system through a one-month deployment and diary study. These studies demonstrate that FETCH is usable and useful, but there is still room for improvement.

Powerful mobile devices with minimal I/O capabilities increase the likelihood that we will want to annex these devices to I/O resources we encounter in the local environment. This opportunistic annexing will require authentication. We present a sensor-based authentication mechanism for mobile devices that relies on physical possession instead of knowledge to setup the initial connection to a public terminal. Our solution provides a simple mechanism for shaking a device to authenticate with the public infrastructure, making few assumptions about the surrounding infrastructure while also maintaining a reasonable level of security.