Smart homes, medical devices, and education systems, among other emerging cyber-physical systems, hold immense promise for sensing-based user interfaces, especially for using fingers and hand gestures as system input. However, vision approaches compatible with time-consuming image processing adopt low 60 Hz location sampling rate (frame rate) for real-time hand gesture recognition. Furthermore, they are not suitable for low-light environment and long detection range. In this paper, we propose RoFin, which first exploits 6 temporal-spatial 2D rolling fingertips for real-time 3D reconstructing of 20-joint hand pose. RoFin designs active optical labeling for finger identification and enhances inside-frame 3D location tracking via high rolling shutter rate (5-8 kHz). These features enable great potentials for enhanced multi-user HCI, virtual writing for Parkinson suffers, etc. We implement RoFin prototypes with wearable gloves attached with low-power single-colored LED nodes and commercial cameras. The experiment results show that (1) In flexible sensing distances up to 2.5 m, RoFin achieves an average labeling parsing accuracy of 85%, (2) In comparison to vision-based techniques, RoFin improves the tracking grain with 4× more sampled points each frame, (3) RoFin reconstructs a hand pose in real time with 16 mm mean deviation error compared with Leap Motion under flexible distance, and (4) we further investigated real-world applications of RoFin, such as air writing with smoothed trajectories and mobile-based letter/number recognition in our developedXameraapp.

This paper introduces MobiChem, a low-cost, portable, practical, and ubiquitous smartphone-based toolkit for fruit monitoring. The key idea is to leverage the light emitted from a smartphone's screen and front camera, coupled with a custom-built screen cover, to perform comprehensive hyperspectral analysis on targeted objects. Specifically, we designed a zero-powered screen cover that selectively filters wavelengths essential for hyperspectral sensing. We then incorporate a CNN-based algorithm and a novel ranking-based learning technique that manipulates the latent space to classify maturity stages and characterize their chemical and physical factors. To demonstrate MobiChem's feasibility, robustness, and practicality, we showcase its application in tomato, banana, and avocado sensing. Our system examines the maturity, chlorophyll, lycopene content, free sugar levels, and firmness, enabling various dietary assessments and food safety applications. Experimental results using 117 tomatoes, 98 bananas, and 73 avocados show MobiChem achieved 95.67% accuracy in chlorophyll concentration measurement, 98.76% for lycopene detection, 93.53% for sugar concentrations analysis, and 91.34% average accuracy in classifying maturity (96.64% for tomato, 86.37% for banana, and 91.03% for avocado).

This paper introduces MobiChem, a low-cost, portable, practical, and ubiquitous smartphone-based toolkit for fruit monitoring. The key idea is to leverage the light emitted from a smartphone's screen and front camera, coupled with a custom-built screen cover, to perform comprehensive hyperspectral analysis on targeted objects. Specifically, we designed a zero-powered screen cover that selectively filters wavelengths essential for hyperspectral sensing. We then incorporate a CNN-based algorithm and a novel ranking-based learning technique that manipulates the latent space to classify maturity stages and characterize their chemical and physical factors. We showcased its application in tomato, banana, and avocado sensing. Our system examines the maturity, chlorophyll, lycopene content, free sugar levels, and firmness, enabling various dietary assessments and food safety applications.

In this paper, we propose Argus, a wearable add-on system based on stripped-down (i.e., compact, lightweight, low-power, limited-capability) mmWave radars. It is the first to achieve egocentric human mesh reconstruction in a multi-view manner. Compared with conventional frontal-view mmWave sensing solutions, it addresses several pain points, such as restricted sensing range, occlusion, and the multipath effect caused by surroundings. To overcome the limited capabilities of the stripped-down mmWave radars (with only one transmit antenna and three receive antennas), we tackle three main challenges and propose a holistic solution, including tailored hardware design, sophisticated signal processing, and a deep neural network optimized for high-dimensional complex point clouds. Extensive evaluation shows that Argus achieves performance comparable to traditional solutions based on high-capability mmWave radars, with an average vertex error of 6.5 cm, solely using stripped-down radars deployed in a multi-view configuration. It presents robustness and practicality across conditions, such as with unseen users and different host devices.

Soil monitoring plays an essential role in agricultural systems. Rather than deploying sensors' antennas above the ground, burying them in the soil is an attractive way to retain a non-intrusive aboveground space. Low Power Wide-Area Network (LPWAN) has shown its long-distance and low-power features for aboveground Internet-of-Things (IoT) communication, presenting a potential of extending to underground cross-soil communication over a wide area, which however has not been investigated before. The variation of soil conditions brings significant signal polarization misalignment, degrading communication reliability. In this paper, we propose Demeter, a low-cost low-power programmable antenna design to keep reliable cross-soil communication automatically. First, we propose a hardware architecture to enable polarization adjustment on commercial-off-the-shelf (COTS) single-RF-chain LoRa radio. Moreover, we develop a low-power programmable circuit to obtain polarization adjustment. We further design an energy-efficient heuristic calibration algorithm and an adaptive calibration scheduling method to keep signal polarization alignment automatically. We implement Demeter with a customized PCB circuit and COTS devices. Then, we evaluate its performance in various soil types and environmental conditions. The results show that Demeter can achieve up to 11.6 dB SNR gain indoors and 9.94 dB outdoors, 4× horizontal communication distance, at least 20 cm deeper underground deployment, and up to 82% energy consumption reduction per day compared with the standard LoRa.

In this paper, we proposed F2Key, the first earable physical security system based on commercial off-the-shelf headphones. F2Key enables impactful applications, such as enhancing voiceprint-based authentication systems, reliable voice assistants, audio deepfake defense, and the legal validity of artifacts. The key idea of F2Key is to establish a stable acoustic sensing field across the user's face and embed the user's facial structures and articulatory habits into a user-specific generative model that serves as a private key. The private key can decrypt the Channel Impulse Response (CIR) profiles provided by the acoustic sensing field into an inferred spectrogram that can match the real one calculated from the corresponding speech, provided that the user's CIR-spectrogram mapping relationship is consistent with the one embedded in the generative model. Extensive experiments demonstrate that F2Key resists 99.9%, 96.4%, and 95.3% of speech replay attacks, mimicry attacks, and hybrid attacks, respectively. We discussed and evaluated F2Key from different perspectives, such as the health consideration and identical twins study, to show the practicality and reliability.

Accurate measurements of soil macronutrients (i.e., nitrogen, phosphorus, and potassium) and moisture play a key role in smart agriculture. However, existing commodity soil sensors are often expensive and the achieved accuracy is unsatisfactory. To address these issues, we present SoilCares, a low-cost soil sensing system enabling accurate and simultaneous monitoring of the concentration levels of soil moisture and macronutrients. SoilCares overcomes key challenges of accommodating diverse soil types and soil textures by introducing a novel membrane-based scheme. For moisture sensing, SoilCares leverages the multi-modal fusion of RF and NIR signals to significantly increase the sensing accuracy. Through delicate hardware design, we enable negligible-cost sensor data transmission using the existing sensing hardware, building up a complete end-to-end soil sensing system. SoilCares is cost-effective ($63.5), portable (0.5 kg), and low-power (236 μW), making it suitable for insitu deployment. On-site experimental results show that SoilCares achieves high macronutrient sensing accuracy with a low RMSE of 0.138, and extremely low moisture estimation error of 1%, outperforming the state-of-the-art research and expensive commodity moisture sensors on the market.

With the advancement of AI-powered personal voice assistants, speaker authentication via earbuds has become increasingly vital, serving as a critical interface between users and mobile devices. However, existing audio-based speaker authentication methods fail to defend against voice spoofing threats such as replay and deep-fake attacks. To counteract these risks, we introduce PiezoBud, a pioneering multi-modal user authentication system that is truly practical and lightweight for earbuds. PiezoBud uses miniature piezoelectric sensors to detect micro-vibrations on the skin, extracting user-specific biometric data to authenticate legitimate access on the local smartphone and protect against malicious attacks. Our exploratory study, involving 85 participants, demonstrates the effectiveness of PiezoBud in various everyday scenarios, including ambient noise, body movement, and in-ear media playing. Using only 15 seconds of enrollment data, PiezoBud achieves an Equal Error Rate (EER) of 1.05% and attain a mean authentication latency of 0.06 seconds on mobile devices. We also evaluate PiezoBud's effectiveness in countering challenging adaptive attack scenarios and its overall performance in various real-world situations. Our evaluation highlights that PiezoBud stands out as a practical, resilient, responsive, and secure option for earbuds users.

Beam selection is a fundamental problem in millimeter-wave (mmWave) communication systems. Yet, most existing beam selection techniques focus on the exploitation of spatial channel features to reduce their airtime overhead in stationary mmWave networks. In this article, we exploit the temporal correlation of wireless channels to facilitate beam selection in mobile mmWave networks. Specifically, we present a temporal beam prediction (TBP) scheme for a mobile mmWave device to predict its future beam direction based on its history beam selection profile. TBP has two challenges in its design: 1) nonuniform history data samples due to the bursty nature of data traffic and 2) nonsmooth beam angles over time due to the multipath effect of channels and the imperfect radiation pattern of phased-array antennas. TBP addresses these two challenges by employing a new mobility-aware LSTM model that takes data timestamp for its training, together with an adversarial learning model to exploit user-independent features for beam steering. We have evaluated TBP through over-the-air (OTA) experiments on a 60-GHz mmWave testbed. Experimental results show that the average prediction error of TBP is less than 7° and that TBP improves the throughput by 60% in representative mmWave networks.

Recent years have shown substantial interest in revealing vulnerability issues of voice-controllable systems on smartphones and smart speakers. While significant prior works have leveraged inaudible signals to attack these smart devices, smart earbuds present unique challenges and vulnerabilities due to their extreme hardware constraints. In this paper, we present EchoAttack, a practical inaudible attack system for smart earbuds. The primary innovation of EchoAttack is the ability to leverage both indirect and direct paths to attack smart earbuds. To search for the optimal path, we design a path-searching algorithm based on the attenuation model of ultrasound. We also propose a novel approach to remove harmonics noise, which improves the attacking signal's SNR further. Finally, we propose using Zigbee radios to sniff the Bluetooth signal and enable a hidden feedback channel without the victim's awareness. We implement the EchoAttack prototype using off-the-shelf hardware components and evaluate the prototypes in four typical indoor and outdoor scenarios using six smart earbuds. Experimental results show that EchoAttack outperforms the pure direct-path attack by 75.8% on average in terms of attack success rate.

Smart homes, medical devices, and education systems, among other emerging cyber-physical systems, hold immense promise for sensing-based user interfaces, especially for using fingers and hand gestures as system input. However, vision approaches compatible with time-consuming image processing adopt low 60 Hz location sampling rate (frame rate) for real-time hand gesture recognition. Furthermore, they are not suitable for low-light environment and long detection range. In this paper, we propose RoFin, which first exploits 6 temporal-spatial 2D rolling fingertips for real-time 3D reconstructing of 20-joint hand pose. RoFin designs active optical labeling for finger identification and enhances inside-frame 3D location tracking via high rolling shutter rate (5--8 KHz). These features enable great potentials for enhanced multi-user HCI, virtual writing for Parkinson suffers, etc. We implement RoFin prototypes with wearable gloves attached with low-power single-colored LED nodes and commercial cameras. The experiment results show that (1) In flexible sensing distances up to 2.5 m, RoFin achieves an average labeling parsing accuracy of 85%. (2) In comparison to vision-based techniques, RoFin improves the tracking grain with 4× more sampled points each frame. (3) RoFin reconstructs a hand pose in real time with 16 mm mean deviation error compared with Leap Motion under flexible distance.

This paper presents EMGSense, a low-effort self-supervised domain adaptation framework for sensing applications based on Electromyography (EMG). EMGSense addresses one of the fundamental challenges in EMG cross-user sensing—the significant performance degradation caused by time-varying biological heterogeneity—in a low-effort (data-efficient and label-free) manner. To alleviate the burden of data collection and avoid labor-intensive data annotation, we propose two EMG-specific data augmentation methods to simulate the EMG signals generated in various conditions and scope the exploration in label-free scenarios. We model combating biological heterogeneity-caused performance degradation as a multi-source domain adaptation problem that can learn from the diversity among source users to eliminate EMG heterogeneous biological features. To relearn the target-user-specific biological features from the unlabeled data, we integrate advanced self-supervised techniques into a carefully designed deep neural network (DNN) structure. The DNN structure can seamlessly perform two training stages that complement each other to adapt to a new user with satisfactory performance. Comprehensive evaluations on two sizable datasets collected from 13 participants indicate that EMGSense achieves an average accuracy of 91.9% and 81.2% in gesture recognition and activity recognition, respectively. EMGSense outperforms the state-of-the-art EMG-oriented domain adaptation approaches by 12.5%-17.4% and achieves a comparable performance with the one trained in a supervised learning manner.

While multi-modal deep learning is useful in distributed sensing tasks like human tracking, activity recognition, and audio and video analysis, deploying state-of-the-art multi-modal models in a wirelessly networked sensor system poses unique challenges. The data sizes for different modalities can be highly asymmetric (e.g., video vs. audio), and these differences can lead to significant delays between streams in the presence of wireless dynamics. Therefore, a slow stream can significantly slow down a multi-modal inference system in the cloud, leading to either increased latency (when blocked by the slow stream) or degradation in inference accuracy (if inference proceeds without waiting). In this paper, we introduce speculative inference on multi-modal data streams to adapt to these asymmetries across modalities. Rather than blocking inference until all sensor streams have arrived and been temporally aligned, we impute any missing, corrupt, or partially-available sensor data, then generate a speculative inference using the learned models and imputed data. A rollback module looks at the class output of speculative inference and determines whether the class is sufficiently robust to incomplete data to accept the result; if not, we roll back the inference and update the model's output. We implement the system in three multi-modal application scenarios using public datasets. The experimental results show that our system achieves 7 -- 128× latency speedup with the same accuracy as six state-of-the-art methods.

We propose a compact noninvasive glucose monitoring system using polarized light, where a user simply needs to place her palm on the device for measuring her current glucose concentration level. The primary innovation of our system is the ability to minimize light scattering from the skin and extract weak changes in light polarization to estimate glucose concentration, all using low-cost hardware. Our system exploits multiple wavelengths and light intensity levels to mitigate the effect of user diversity and confounding factors (e.g., collagen and elastin in the dermis). It then infers glucose concentration using a generic learning model, thus no additional calibration is needed. We design and fabricate a compact (17 cm x 10 cm x 5 cm) and low-cost (i.e., <$250) prototype using off-the-shelf hardware. We evaluate our system with 41 diabetic patients and 9 healthy participants. In comparison to a continuous glucose monitor approved by U.S. Food and Drug Administration (FDA), 89% of our results are within zone A (clinically accurate) of the Clarke Error Grid. The absolute relative difference (ARD) is 10%. The r and p values of the Pearson correlation coefficients between our predicted glucose concentration and reference glucose concentration are 0.91 and 1.6 x 10-143, respectively. These errors are comparable with FDA-approved glucose sensors, which achieve ≈90% clinical accuracy with a 10% mean ARD.

This paper presents a battery-free wearable eye tracker that tracks both the 2D position and diameter of a pupil based on its light absorption property. With a few near-infrared (NIR) lights and photodiodes around the eye, NIR lights sequentially illuminate the eye from various directions while photodiodes sense spatial patterns of reflected light, which are used to infer pupil's position and diameter on the fly via a lightweight inference algorithm. The system also exploits characteristics of different eye movement stages and adjusts its sensing and computation accordingly for further energy savings. A prototype is built with off-the-shelf hardware components and integrated into a regular pair of glasses. Experiments with 22 participants show that the system achieves 0.8-mm mean error in tracking pupil position (2.3 mm at the 95th percentile) and 0.3-mm mean error in tracking pupil diameter (0.9 mm at the 95th percentile) at 120-Hz output frame rate, consuming 395 µW mean power supplied by two small, thin solar cells on glasses side arms.

We present a self-powered module for gesture recognition that utilizes small, low-cost photodiodes for both energy harvesting and gesture sensing. Operating in the photovoltaic mode, photodiodes harvest energy from ambient light. In the meantime, the instantaneously harvested power from individual photodiodes is monitored and exploited as clues for sensing finger gestures in proximity. Harvested power from all photodiodes are aggregated to drive the whole gesture-recognition module including the micro-controller running the recognition algorithm. We design robust, lightweight algorithm to recognize finger gestures in the presence of ambient light fluctuations. We fabricate two prototypes to facilitate user's interaction with smart glasses and smart watch. Results show 99.7%/98.3% overall precision/recall in recognizing five gestures on glasses and 99.2%/97.5% precision/recall in recognizing seven gestures on the watch. The system consumes 34.6 µW/74.3 µW for the glasses/watch and thus can be powered by the energy harvested from ambient light. We also test system's robustness under varying light intensities, light directions, and ambient light fluctuations, where the system maintains high recognition accuracy (> 96%) in all tested settings.

demonstration Public Access Share on Demo: Ultra-Low Power Gaze Tracking for Virtual Reality Authors: Tianxing Li Dartmouth College, Hanover, NH, USA Dartmouth College, Hanover, NH, USAView Profile , Emmanuel S. Akosah Dartmouth College, Hanover, NH, USA Dartmouth College, Hanover, NH, USAView Profile , Qiang Liu Dartmouth College, Hanover, NH, USA Dartmouth College, Hanover, NH, USAView Profile , Xia Zhou Dartmouth College, Hanover, NH, USA Dartmouth College, Hanover, NH, USAView Profile Authors Info & Claims MobiCom '17: Proceedings of the 23rd Annual International Conference on Mobile Computing and NetworkingOctober 2017 Pages 490–492https://doi.org/10.1145/3117811.3119866Published:04 October 2017Publication History 1citation202DownloadsMetricsTotal Citations1Total Downloads202Last 12 Months25Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

We present LiGaze, a low-power approach to gaze tracking tailored to VR. It relies on a few low-cost photodiodes, eliminating the need for cameras and active infrared emitters. Reusing light emitted from the VR screen, LiGaze leverages photodiodes around a VR lens to measure reflected screen light in different directions. It then infers gaze direction by exploiting pupil's light absorption property. The core of LiGaze is to deal with screen light dynamics and extract changes in reflected light related to pupil movement. We design and fabricate a LiGaze prototype using off-the-shelf photodiodes. Its sensing and computation consume 791μW in total.

Tracking user's eye fixation direction is crucial to virtual reality (VR): it eases user's interaction with the virtual scene and enables intelligent rendering to improve user's visual experiences and save system energy. Existing techniques commonly rely on cameras and active infrared emitters, making them too expensive and power-hungry for VR headsets (especially mobile VR headsets).

We present StarLight, an infrastructure-based sensing system that reuses light emitted from ceiling LED panels to reconstruct fine-grained user skeleton postures continuously in real time. It relies on only a few (e.g., 20) photodiodes placed at optimized locations to passively capture low-level visual clues (light blockage information), with neither cameras capturing sensitive images, nor on-body devices, nor electromagnetic interference. It then aggregates the blockage information of a large number of light rays from LED panels and identifies best-fit 3D skeleton postures. StarLight greatly advances the prior light-based sensing design by dramatically reducing the number of intrusive sensors, overcoming furniture blockage, and supporting user mobility. We build and deploy StarLight in a 3.6 m x 4.8 m office room, with customized 20 LED panels and 20 photodiodes. Experiments show that StarLight achieves 13.6 degree mean angular error for five body joints and reconstructs a mobile skeleton at a high frame rate (40 FPS). StarLight enables a new unobtrusive sensing paradigm to augment today's mobile sensing for continuous and accurate behavioral monitoring.

Pervasive Wi-Fi connectivity is attractive for users in places not covered by cellular services (e.g., when traveling abroad). However, the power drain of frequent Wi-Fi scans undermines the device's battery life, preventing users from staying always connected and fetching synced emails and instant message notifications (e.g., WhatsApp). We study the energy overhead of scan and roaming in detail and refer to it as the scan tax problem. Our findings show that the main processor is the primary culprit of the energy overhead. We propose a simple and effective architectural change of offloading scans to the Wi-Fi radio. We design and build WiScan to fully exploit the gain of scan offloading. Our experiments demonstrate that WiScan achieves 90%+ of the maximal connectivity, while saving 50-62% energy for seeking connectivity.

We present LiSense, the first-of-its-kind system that enables both data communication and fine-grained, real-time human skeleton reconstruction using Visible Light Communication (VLC). LiSense uses shadows created by the human body from blocked light and reconstructs 3D human skeleton postures in real time. We overcome two key challenges to realize shadow-based human sensing. First, multiple lights on the ceiling lead to diminished and complex shadow patterns on the floor. We design light beacons enabled by VLC to separate light rays from different light sources and recover the shadow pattern cast by each individual light. Second, we design an efficient inference algorithm to reconstruct user postures using 2D shadow information with a limited resolution collected by photodiodes embedded in the floor. We build a 3 m x 3 m LiSense testbed using off-the-shelf LEDs and photodiodes. Experiments show that LiSense reconstructs the 3D user skeleton at 60 Hz in real time with 10 degrees mean angular error for five body joints.

We present HiLight, a new form of real-time screen-camera communication without showing any coded images (e.g., barcodes) for off-the-shelf smart devices. HiLight encodes data into pixel translucency change atop any screen content, so that camera-equipped devices can fetch the data by turning their cameras to the screen. HiLight leverages the alpha channel, a well-known concept in computer graphics, to encode bits into the pixel translucency change. By removing the need to directly modify pixel RGB values, HiLight overcomes the key bottleneck of existing designs and enables real-time unobtrusive communication while supporting any screen content. We build a HiLight prototype using off-the-shelf smart devices and demonstrate its efficacy and robustness in practical settings. By offering an unobtrusive, flexible, and lightweight communication channel between screens and cameras, HiLight opens up opportunities for new HCI and context-aware applications, e.g., smart glasses communicating with screens to realize augmented reality.

No abstract available.

Much of the stress and strain of student life remains hidden. The StudentLife continuous sensing app assesses the day-to-day and week-by-week impact of workload on stress, sleep, activity, mood, sociability, mental well-being and academic performance of a single class of 48 students across a 10 week term at Dartmouth College using Android phones. Results from the StudentLife study show a number of significant correlations between the automatic objective sensor data from smartphones and mental health and educational outcomes of the student body. We also identify a Dartmouth term lifecycle in the data that shows students start the term with high positive affect and conversation levels, low stress, and healthy sleep and daily activity patterns. As the term progresses and the workload increases, stress appreciably rises while positive affect, sleep, conversation and activity drops off. The StudentLife dataset is publicly available on the web.

We present HiLight, a new form of screen-camera communication without the need of any coded images (e.g. barcodes) for off-the-shelf smart devices. HiLight hides information underlying any images shown on a LED or an OLED screen, so that camera-equipped smart devices can fetch the information by turning their cameras to the screen. HiLight achieves this by leveraging the orthogonal transparency (alpha) channel, a well-known concept in computer graphics, to embed bits into pixel translucency changes without the need of modifying pixel color values. We demonstrated HiLight's feasibility using smartphones. By offering an unobtrusive, flexible, and lightweight communication channel between screens and cameras, HiLight opens up opportunities for new HCI and context-aware applications to emerge, e.g. smart glass communicates with screens for additional personalized information to realize augmented reality.

Existing dual image deblurring methods usually model blurred image pairs being taken from exactly the same viewpoint and restore a single clear image. This imposes a strong assumption that the latent clear images of both images must be completely identical. In contrast to this restricted scenario, we assume that the restored pair are different, but can be approximated by image warping due to small viewpoint change. This allows us to deblur each image individually, but still being able to make use of the matched areas in image pairs. Our deblurring algorithm iteratively performs a two-directional dual image deblurring, which uses the Split Bregman method, and matches the latent clear image pairs by a homography. Experiments show that the proposed algorithm automatically recovers clear images from blurred image pairs in the same scene. Statistics suggest that the method is robust to viewpoint change and different noise levels.