Fine-grained inspection of ferrous structures, such as steel rebars and iron pipes, is essential for ensuring structural health/integrity. However, existing non-destructive imaging techniques often suffer from coarse spatial resolution, high operational costs, and limited mobility support, hence severely restricting their practical deployment. For example, ground-penetrating radar (GPR), constrained by its operating wavelength, cannot resolve sub-centimeter features or recover fine contours of embedded ferrous structures.

As the number of smart devices increases in our lives, the data they collect to perform valuable tasks, such as voice assistant requests, comes at the cost of user privacy. To mitigate their privacy impact, emerging usable privacy-aware sensing (UPAS) research has relied on cross-disciplinary approaches that extend past the core focus of broader academic research communities, such as Security & Privacy or Human-Computer Interaction. These works incorporate privacy design principles, whereby systems include safeguards by combining usable privacy (UP) with privacy-aware sensing (PAS) design to protect users' privacy. To better understand this emerging area of research, we conducted a mixed qualitative and quantitative Systematization of Knowledge (SoK). With a thorough review of pertinent literature, resulting in 114 selected works (reduced from 10,122 across 12 venues), we found that, despite the similarity of these works, many are dispersed across multiple communities, utilize community-specific jargon and keywords, and minimally overlap in design and evaluation approaches, potentially hindering cross-pollination across communities and thereby slowing the growth of this emerging research area. Thus, these factors helped reveal a research gap in this space. We use these findings to present four research themes and provide community and design recommendations to encourage cross-disciplinary UPAS research.

Fiducial marking is indispensable in mobile robots, including their pose calibration, contextual perception, and navigation. However, existing fiducial markers rely solely on vision-based perception which suffers such limitations as occlusion, energy overhead, and privacy leakage.

Internet-connected cameras support many useful home monitoring and health applications. However, these same cameras indiscriminately capture sensitive and Personally Identifiable Information (PII), limiting their acceptance in certain settings, such as the home. Prior works removed Region of Interest (ROI) to secure images and improve privacy. However, the methods that rely solely on RGB information to find persons are susceptible to environmental and lighting conditions, causing them to fail and leak PII. From our deployment study, nearly half of the images containing persons had a PII leakage when using RGB-only methods. Furthermore, ROI removal is often performed off-device, requiring the server performing these operations to be trustworthy. This work presents the PrivacyLens system, where with the addition of thermal sensing, our system has a significantly enhanced ability to find persons in RGB images and video and efficiently remove them on the device before any data is stored or transmitted, all while staying under typical IoT power constraints. From our aforementioned deployment study in an office-building atrium, family home, and outdoor park environment, the PrivacyLens prototype effectively removes PII with a sanitization rate of 99.1%. Additionally, PrivacyLens can use its embedded GPU to generate on-device features for downstream CV/ML tasks, as shown in three illustrative applications, further reducing the collection and storage of PII.

Enabling computing systems to understand user interactions with everyday surfaces and objects can drive a wide range of applications. However, existing vibration-based sensors (e.g., accelerometers) lack the sensitivity to detect light touch gestures or the bandwidth to recognize activity containing high-frequency components. Conversely, microphones are highly susceptible to environmental noise, degrading performance. Each time an object impacts a surface, Surface Acoustic Waves (SAWs) are generated that propagate along the air-to-surface boundary. This work repurposes a Voice PickUp Unit (VPU) to capture SAWs on surfaces (including smooth surfaces, odd geometries, and fabrics) over long distances and in noisy environments. Our custom-designed signal acquisition, processing, and machine learning pipeline demonstrates utility in both interactive and activity recognition applications, such as classifying trackpad-style gestures on a desk and recognizing 16 cooking-related activities, all with >97% accuracy. Ultimately, SAWs offer a unique signal that can enable robust recognition of user touch and on-surface events.

Road surface markings, like symbols and line markings, are vital traffic infrastructures for driving safety and efficiency. However, real-world conditions can impair the utility of existing road markings. For example, adverse weather conditions such as snow and rain can quickly obliterate visibility.

Touchscreen devices, designed with an assumed range of user abilities and interaction patterns, often present challenges for individuals with diverse abilities to operate independently. Prior efforts to improve accessibility through tools or algorithms necessitated alterations to touchscreen hardware or software, making them inapplicable for the large number of existing legacy devices. In this paper, we introduce BrushLens, a hardware interaction proxy that performs physical interactions on behalf of users while allowing them to continue utilizing accessible interfaces, such as screenreaders and assistive touch on smartphones, for interface exploration and command input. BrushLens maintains an interface model for accurate target localization and utilizes exchangeable actuators for physical actuation across a variety of device types, effectively reducing user workload and minimizing the risk of mistouch. Our evaluations reveal that BrushLens lowers the mistouch rate and empowers visually and motor impaired users to interact with otherwise inaccessible physical touchscreens more effectively.

Magnetic sensing is emerging as an enabling technology for various engaging applications. Representative use cases include high-accuracy posture tracking, human-machine interaction, and haptic sensing. This technology uses multiple MEMS magnetometers to capture the changing magnetic field at a close distance. However, magnetometers are susceptible to real-world disturbances, such as hard- and soft-iron effects. As a result, users need to perform a cumbersome and lengthy calibration process frequently, severely limiting the usability of magnetic tracking.

Pervasive and interactive displays promise to present our digital content seamlessly throughout our environment. However, traditional display technologies do not scale to room-wide applications due to high per-unit-area costs and the need for constant wired power and data infrastructure. This research proposes the use of photochromic paint as a display medium. Applying the paint to any surface or object creates ultra-low-cost displays, which can change color when exposed to specific wavelengths of light. We develop new paint formulations that enable wide area application of photochromic material. Along with a specially modified wide-area laser projector and depth camera that can draw custom images and create on-demand, room-wide user interfaces on photochromic enabled surfaces. System parameters such as light intensity, material activation time, and user readability are examined to optimize the display. Results show that images and user interfaces can last up to 16 minutes and can be updated indefinitely. Finally, usage scenarios such as displaying static and dynamic images, ephemeral notifications, and the creation of on-demand interfaces, such as light switches and music controllers, are demonstrated and explored. Ultimately, the UbiChromics system demonstrates the possibility of extending digital content to all painted surfaces.

Sound presents an invaluable signal source that enables computing systems to perform daily activity recognition. However, microphones are optimized for human speech and hearing ranges: capturing private content, such as speech, while omitting useful, inaudible information that can aid in acoustic recognition tasks. We simulated acoustic recognition tasks using sounds from 127 everyday household/workplace objects, finding that inaudible frequencies can act as a substitute for privacy-sensitive frequencies. To take advantage of these inaudible frequencies, we designed a Raspberry Pi-based device that captures inaudible acoustic frequencies with settings that can remove speech or all audible frequencies entirely. We conducted a perception study, where participants “eavesdropped’’ on PrivacyMic’s filtered audio and found that none of our participants could transcribe speech. Finally, PrivacyMic’s real-world activity recognition performance is comparable to our simulated results, with over 95% classification accuracy across all environments, suggesting immediate viability in performing privacy-preserving daily activity recognition.

Accurate tracking of the hands and fingers allows users to employ natural gestures in various interactive applications. Hand tracking also supports health applications, such as monitoring face-touching, a common vector for infectious disease. However, for both types of applications, the utility of hand tracking is often limited by the impracticality of bulky tethered systems (e.g., instrumented gloves) or inherent limitations (e.g., Line of Sight or privacy concerns with vision-based systems). These limitations have severely restricted the adoption of hand tracking in real-world applications. We present MagX, a fully untethered on-body hand tracking system utilizing passive magnets and a novel magnetic sensing platform. Since passive magnets require no maintenance, they can be worn on the hands indefinitely, and only the sensor board needs recharging, akin to a smartwatch.

Accurate tracking of the hands and fingers allows users to employ natural gestures in various interactive applications, e.g., controller-free interaction in augmented reality. Hand tracking also supports health applications, such as monitoring face-touching, a common vector for infectious disease. However, for both types of applications, the utility of hand tracking is often limited by the impracticality of bulky tethered systems (e.g., instrumented gloves) or inherent limitations (e.g., Line of Sight or privacy concerns with vision-based systems). These limitations have severely restricted the adoption of hand tracking in real-world applications. We demonstrate MagX, a fully untethered on-body hand tracking system utilizing passive magnets and a novel magnetic sensing platform. Since passive magnets require no maintenance, they can be worn on the hands indefinitely, and only the sensor board needs recharging, akin to a smartwatch.

Smart speakers with voice agents are becoming increasingly common. However, the agent's voice always emanates from the device, even when that information is contextually and spatially relevant elsewhere. Digital Ventriloquism allows smart speakers to render sound onto everyday objects, such that it appears they are speaking and are interactive. This can be achieved without any modification of objects or the environment. For this, we used a highly directional pan-tilt ultrasonic array. By modulating a 40 kHz ultrasonic signal, we can emit sound that is inaudible "in flight" and demodulates to audible frequencies when impacting a surface through acoustic parametric interaction. This makes it appear as though the sound originates from an object and not the speaker. We ran a study in which we projected speech onto five objects in three environments, and found that participants were able to correctly identify the source object 92% of the time and correctly repeat the spoken message 100% of the time, demonstrating our digital ventriloquy is both directional and intelligible.

BeamBand is a wrist-worn system that uses ultrasonic beamforming for hand gesture sensing. Using an array of small transducers, arranged on the wrist, we can ensem-ble acoustic wavefronts to project acoustic energy at spec-ified angles and focal lengths. This allows us to interro-gate the surface geometry of the hand with inaudible sound in a raster-scan-like manner, from multiple view-points. We use the resulting, characteristic reflections to recognize hand pose at 8 FPS. In our user study, we found that BeamBand supports a six-class hand gesture set at 94.6% accuracy. Even across sessions, when the sensor is removed and reworn later, accuracy remains high: 89.4%. We describe our software and hardware, and future ave-nues for integration into devices such as smartwatches and VR controllers.

Interferi is an on-body gesture sensing technique using acoustic interferometry. We use ultrasonic transducers resting on the skin to create acoustic interference patterns inside the wearer's body, which interact with anatomical features in complex, yet characteristic ways. We focus on two areas of the body with great expressive power: the hands and face. For each, we built and tested a series of worn sensor configurations, which we used to identify useful transducer arrangements and machine learning fea-tures. We created final prototypes for the hand and face, which our study results show can support eleven- and nine-class gestures sets at 93.4% and 89.0% accuracy, re-spectively. We also evaluated our system in four continu-ous tracking tasks, including smile intensity and weight estimation, which never exceed 9.5% error. We believe these results show great promise and illuminate an inter-esting sensing technique for HCI applications.

Robust, wide-area sensing of human environments has been a long-standing research goal. We present Sozu, a new low-cost sensing system that can detect a wide range of events wirelessly, through walls and without line of sight, at whole-building scale. To achieve this in a battery-free manner, Sozu tags convert energy from activities that they sense into RF broadcasts, acting like miniature self-powered radio stations. We describe the results from a series of iterative studies, culminating in a deployment study with 30 instrumented objects. Results show that Sozu is very accurate, with true positive event detection exceeding 99%, with almost no false positives. Beyond event detection, we show that Sozu can be extended to detect richer signals, such as the state, intensity, count, and rate of events.