Gestures are a promising candidate as an input modality for ambient computing where conventional input modalities such as touchscreens are not available. Existing works have focused on gesture recognition using image sensors. However, their cost, high battery consumption, and privacy concerns made cameras challenging as an always-on solution. This paper introduces an efficient gesture recognition technique using a miniaturized 60 GHz radar sensor. The technique recognizes four directional swipes and an omni-swipe using a radar chip (6.5 × 5.0 mm) integrated into a mobile phone. We developed a convolutional neural network model efficient enough for battery powered and computationally constrained processors. Its model size and inference time is less than 1/5000 compared to an existing gesture recognition technique using radar. Our evaluations with large scale datasets consisting of 558,000 gesture samples and 3,920,000 negative samples demonstrated our algorithm’s efficiency, robustness, and readiness to be deployed outside of research laboratories.

ZebraSense is a novel dual-sided woven touch sensor that can recognize and differentiate interactions on the top and bottom surfaces of the sensor. ZebraSense is based on an industrial multi-layer textile weaving technique, yet it enables a novel capacitive sensing paradigm, where each sensing element contributes to touch detection on both surfaces of the sensor simultaneously. Unlike the common "sensor sandwich" approach used in previous work, ZebraSense inherently minimizes the number of sensing elements, which drastically simplifies both sensor construction and its integration into soft goods, while preserving maximum sensor resolution. The experimental evaluation confirmed the validity of our approach and demonstrated that ZebraSense is a reliable, efficient, and accurate solution for detecting user gestures in various dual-sided interaction scenarios, allowing for new use cases in smart apparel, home decoration, toys, and other textile objects.

Today we can see an increasing number of object recognition systems of very different sizes, portability, embedability and form factors which are starting to become part of the ubiquitous, tangible, mobile and wearable computing ecosystems that we might make use of in our daily lives. These systems rely on a variety of technologies including computer vision, radar, acoustic sensing, tagging and smart objects.

This paper presents Soli , a new, robust, high-resolution, low-power, miniature gesture sensing technology for human-computer interaction based on millimeter-wave radar. We describe a new approach to developing a radar-based sensor optimized for human-computer interaction, building the sensor architecture from the ground up with the inclusion of radar design principles, high temporal resolution gesture tracking, a hardware abstraction layer (HAL), a solid-state radar chip and system architecture, interaction models and gesture vocabularies, and gesture recognition. We demonstrate that Soli can be used for robust gesture recognition and can track gestures with sub-millimeter accuracy, running at over 10,000 frames per second on embedded hardware.

We present a wearable system that uses ambient electromagnetic interference (EMI) as a signature to identify electronic devices and support proxemic interaction. We designed a low cost tool, called EMI Spy, and a software environment for rapid deployment and evaluation of ambient EMI-based interactive infrastructure. EMI Spy captures electromagnetic interference and delivers the signal to a user's mobile device or PC through either the device's wired audio input or wirelessly using Bluetooth. The wireless version can be worn on the wrist, communicating with the user;s mobile device in their pocket. Users are able to train the system in less than 1 second to uniquely identify displays in a 2-m radius around them, as well as to detect pointing at a distance and touching gestures on the displays in real-time. The combination of a low cost EMI logger and an open source machine learning tool kit allows developers to quickly prototype proxemic, touch-to-connect, and gestural interaction. We demonstrate the feasibility of mobile, EMI-based device and gesture recognition with preliminary user studies in 3 scenarios, achieving 96% classification accuracy at close range for 6 digital signage displays distributed throughout a building, and 90% accuracy in classifying pointing gestures at neighboring desktop LCD displays. We were able to distinguish 1- and 2-finger touching with perfect accuracy and show indications of a way to determine power consumption of a device via touch. Our system is particularly well-suited to temporary use in a public space, where the sensors could be distributed to support a popup interactive environment anywhere with electronic devices. By designing for low cost, mobile, flexible, and infrastructure-free deployment, we aim to enable a host of new proxemic interfaces to existing appliances and displays

We present a multimodal on-surface and near-surface sensing technique for planar, curved and flexible surfaces. Our technique leverages temporal multiplexing of signals coming from a universal interdigitated electrode design, which is printed as a single conductive layer on a flexible substrate. It supports sensing of touch and proximity input, and moreover is capable of capturing several levels of pressure and flexing. We leverage recent developments in conductive inkjet printing as a way to prototype electrode patterns, and combine this with our hardware module for supporting the full range of sensing methods. As the technique is low-cost and easy to implement, it is particularly well-suited for prototyping touch- and hover-based user interfaces, including curved and deformable ones.

WristQue combines environmental and inertial sensing with precise indoor localization into a wristband wearable device that serves as the user's personal control interface to networked infrastructure. WristQue enables users to take control of devices around them by pointing to select and gesturing to control. At the same time, it uniquely identifies and locates users to deliver personalized automatic control of the user's environment. In this paper, the hardware and software components of the WristQue system are introduced, and a number of applications for lighting and HVAC control are presented, using pointing and gesturing as a new human interface to these networked systems.