Multimodal human action recognition (HAR) utilizes complementary data for activity classification. Built on traditional HAR tasks, recent advances in Large Language Models (LLMs) enable detailed descriptions and causal reasoning of human actions, advancing new tasks of human action understanding (HAU) and human action reasoning (HARn). However, most LLMs, especially multimodal Large Vision-Language Models (LVLMs), struggle with modalities other than RGB images, like depth, IMU, ormmWave, due to a lack of large-scale datasets in these task domains. Existing HAR datasets provide only coarse-grained annotations, in-sufficient for depicting the detailed action dynamics required in HAU and HARn tasks. Simply combining annotations and generating captions with LLMs often lacks necessary logical and spatiotemporal consistency. In this paper, we introduce CUHK-X, a large-scale multi-modal dataset and benchmarks for HAR, HAU, and HARn. It includes 64,267 samples of 40 actions performed by 30 participants across two indoor environments, covering diverse daily scenarios. To address the challenge of spatiotemporal inconsistencies in captions, we propose a prompt-based scene creation method that leverages LLMs to generate logically connected activity sequences. CUHK-X also includes three benchmarks with six tasks to evaluate state-of-the-art models. Experimental results show average accuracies of 76.52% for HAR, 40.76% for HAU, and 70.25% for HARn. This large-scale multimodal dataset aims to empower the research community to apply, develop, and adapt data-intensive learning techniques for a wide range of human activity-related tasks.

Despite the global mental health crisis, access to screenings, professionals, and treatments remains high. In collaboration with licensed psychotherapists, we propose a C onversational AI T herapist with psychotherapeutic I nterventions (CaiTI), a platform that leverages large language models (LLMs) and smart devices to enable better mental health self-care. CaiTI can screen the day-to-day functioning using natural and psychotherapeutic conversations. CaiTI leverages reinforcement learning to provide personalized conversation flow. CaiTI can accurately understand and interpret user responses. When the user needs further attention during the conversation, CaiTI can provide conversational psychotherapeutic interventions, including cognitive behavioral therapy and motivational interviewing. Leveraging the datasets prepared by the licensed psychotherapists, we experiment and microbenchmark various LLMs’ performance in tasks along CaiTI’s conversation flow and discuss their strengths and weaknesses. With the psychotherapists, we implement CaiTI and conduct 14-day and 24-week studies. The study results, validated by therapists, demonstrate that CaiTI can converse with users naturally, accurately understand and interpret user responses, and provide psychotherapeutic interventions appropriately and effectively. We showcase the potential of CaiTI LLMs to assist the mental therapy diagnosis and treatment and improve day-to-day functioning screening and precautionary psychotherapeutic intervention systems.

Large Language Models (LLMs) have shown immense human-like capabilities for reasoning and generating digital content. However, their ability to freely sense, interact, and actuate the physical domain remains significantly limited due to three fundamental challenges: (1) physical environments require specialized sensors for different tasks, yet deploying dedicated sensors for each application is impractical; (2) events and objects of interest are often localized to small areas within large spaces, making them difficult to detect with static sensor networks; and (3) foundation models need flexible actuation capabilities to meaningfully interact with the physical world. To bridge this gap, we introduce EmbodiedFly, an embodied LLM agent combining a foundation model pipeline with a reconfigurable drone platform to observe, understand, and interact with the physical world. Our co-design approach features (1) a FM orchestration framework connecting multiple LLMs, VLMs, and an open-set object detection model; (2) a novel image segmentation technique that identifies task-relevant areas; and (3) a custom drone platform that autonomously reconfigures with appropriate sensors and actuators based on commands from the FM orchestration framework. Through real-world deployments, we demonstrate that EmbodiedFly completes diverse physical tasks with up to \(85\%\) higher success rates compared to traditional approaches leveraging static deployments.

Ultra-wide fisheye cameras (FoV > 180°) offer unmatched scene coverage but introduce severe geometric distortions that degrade vision system performance. Existing rectification methods struggle with such extreme FoVs due to the inherent limitations of single-perspective projections and the scarcity of ground truth data. We propose a novel split-and-combine framework that rectifies ultra-wide fisheye images into 5-face cubemaps. Our approach begins with a lightweight CNN estimating geometric parameters to guide a structured decomposition of the fisheye image into directional regions. Each region is independently corrected using a two-stage pipeline: a flow-based pre-corrector for geometric warping and a diffusion-based enhancer for detail restoration. We train on a synthetic dataset derived from real-world perspective images, enabling partial supervision. Preliminary results demonstrate strong quantitative and qualitative performance, validating our modular architecture as an effective solution for high-fidelity rectification in ultra-wide fisheye imagery.

Recent advances in Large Language Models (LLMs) have propelled intelligent agents from reactive responses to proactive support. While promising, existing proactive agents either rely exclusively on observations from enclosed environments (e.g., desktop UIs) with direct LLM inference or employ rule-based proactive notifications, leading to suboptimal user intent understanding and limited functionality for proactive service. In this paper, we introduce ContextAgent, the first context-aware proactive agent that incorporates extensive sensory contexts surrounding humans to enhance the proactivity of LLM agents. ContextAgent first extracts multi-dimensional contexts from massive sensory perceptions on wearables (e.g., video and audio) to understand user intentions. ContextAgent then leverages the sensory contexts and personas from historical data to predict the necessity for proactive services. When proactive assistance is needed, ContextAgent further automatically calls the necessary tools to assist users unobtrusively. To evaluate this new task, we curate ContextAgentBench, the first benchmark for evaluating context-aware proactive LLM agents, covering 1,000 samples across nine daily scenarios and twenty tools. Experiments on ContextAgentBench show that ContextAgent outperforms baselines by achieving up to 8.5% and 6.0% higher accuracy in proactive predictions and tool calling, respectively. We hope our research can inspire the development of more advanced, human-centric, proactive AI assistants. The code and dataset are publicly available at https://github.com/openaiotlab/ContextAgent.

Cardio exercise elevates both heart rate and respiration rate, resulting in distinct physiological changes that affect speech patterns, pitch, breathing sounds, and heart sounds. These variations, which occur post-exercise, are influenced by factors such as exercise intensity and individual fitness levels. A comprehensive audio dataset is critically needed to capture post-exercise physiological changes, as existing datasets focus mainly on resting speech, breathing, and heart sounds, neglecting the dynamic shifts following physical exertion. Current datasets fail to capture unique post-exercise variations like speech disfluencies, altered breathing patterns, and variable heart sound intensities, limiting model generalizability to post-exercise conditions. To address this gap, we recruited 59 subjects from diverse backgrounds to engage in cardio exercise, specifically running, reaching varied exertion levels to produce a rich dataset. Our dataset includes 250 sessions totaling 143 minutes of structured reading, 47 minutes of spontaneous speech, 71 minutes of breathing sounds, and 62.5 minutes of phonocardiogram (PCG) recordings. We designed and deployed preliminary case studies to show that speech changes post-cardio could serve as an indicator of exertion level. We envision this dataset as a foundational resource for designing models in speech and cardiorespiratory monitoring that are resilient to the physiological shifts induced by exercise. This dataset could advance natural language processing (NLP) applications, mobile health, and wearable sensing technologies by enabling resilient and accurate physiological monitoring in real-world conditions.

Foundation models (FM) have shown immense human-like capabilities for generating digital media. However, foundation models that can freely sense, interact, and actuate the physical domain is far from being realized. This is due to 1) requiring dense deployments of sensors to fully cover and analyze large spaces, while 2) events often being localized to small areas, making it difficult for FMs to pinpoint relevant areas of interest relevant to the current task. We propose FlexiFly, a platform that enables FMs to "zoom in" and analyze relevant areas with higher granularity to better understand the physical environment and carry out tasks. FlexiFly accomplishes by introducing 1) a novel image segmentation technique that aids in identifying relevant locations and 2) a modular and reconfigurable sensing and actuation drone platform that FMs can actuate to "zoom in" with relevant sensors and actuators. We demonstrate through real smart home deployments that FlexiFly enables FMs and LLMs to complete diverse tasks up to 85% more successfully. FlexiFly is critical step towards FMs and LLMs that can naturally interface with the physical world.

Dialogue systems are designed to offer human users social support or functional services through natural language interactions. Traditional conversation research has put significant emphasis on a system’s response-ability, including its capacity to understand dialogue context and generate appropriate responses. However, the key element of proactive behavior—a crucial aspect of intelligent conversations—is often overlooked in these studies. Proactivity empowers conversational agents to lead conversations towards achieving pre-defined targets or fulfilling specific goals on the system side. Proactive dialogue systems are equipped with advanced techniques to handle complex tasks, requiring strategic and motivational interactions, thus representing a significant step towards artificial general intelligence. Motivated by the necessity and challenges of building proactive dialogue systems, we provide a comprehensive review of various prominent problems and advanced designs for implementing proactivity into different types of dialogue systems, including open-domain dialogues, task-oriented dialogues, and information-seeking dialogues. We also discuss real-world challenges that require further research attention to meet application needs in the future, such as proactivity in dialogue systems that are based on large language models, proactivity in hybrid dialogues, evaluation protocols and ethical considerations for proactive dialogue systems. By providing a quick access and overall picture of the proactive dialogue systems domain, we aim to inspire new research directions and stimulate further advancements towards achieving the next level of conversational AI capabilities, paving the way for more dynamic and intelligent interactions within various application domains.

Large Language Models (LLMs) have shown significant potential in performing inferences on various tasks using heterogeneous sensors with minimal human intervention. Despite their promise, challenges such as high inference overhead and limitations on resource-constrained edge devices remain. Additionally, model hallucinations, particularly those arising from cognitive biases when interpreting numerical data, hinder performance. This work introduces a novel technique, embedding interpolation, to enhance LLMs' understanding of sensor measurements and mitigate inference overhead on edge devices. By computing embeddings through pre-computed boundary embeddings instead of directly from the input, we improve efficiency and accuracy. The effective-ness of this approach is demonstrated through visualizations with image generation models.

In complex environments where visibility is severely compromised, such as smoke-filled areas or dense forests, traditional single-sensor systems often fail to provide accurate and reliable data for robots to autonomously navigate and avoid obstacles effectively, posing significant operational challenges and safety risks. Ground-based sensing platforms are further limited by their restricted mobility, hindering access to remote or hazardous areas. Multimodal sensing, which combines various sensor technologies, offers a robust solution to these challenges. Drones, with their high maneuverability and ability to reach inaccessible locations, can capture detailed data from varied altitudes and perspectives. In this demonstration, we introduce SPECTRA, a drone-based multispectral sensing platform that combines thermal cameras, LiDAR, mmWave, and RGB cameras to reliably perform tasks in challenging environments. SPECTRA is a software-hardware co-design platform that can enable and fuse different sensors locally based on the dynamic environment and interpret user tasks using Large Language Models (LLM). We demonstrate SPECTRA's ability to explore and execute assigned tasks in complex environments.

Over half a trillion recreational runners worldwide engage in running for psychological, health, and social benefits. Running metrics are essential for motivation, goal setting, performance improvement, health management, and injury prevention. Although wearable devices like fitness trackers and smartwatches offer various metrics, they often perform poorly on treadmills and can be uncomfortable or restrictive. In this work, we propose a non-contact, real-time, smartphone-based approach to estimate running metrics, including cadence, ground contact time (GCT), and balance, using the sound produced during treadmill running. In collaboration with a licensed running coach, we recruited over 50 subjects with varying levels of running expertise. We collected treadmill running sounds and ground-truth running metrics in different environments. We designed and developed a multi-task learning (MTL) machine learning mobile system to capture the treadmill running sounds and estimate running metrics in situ. Our proposed method shows comparable accuracy in estimating running metrics to commercial off-the-shelf (COTS) wearable devices.

Speech super-resolution techniques offer a promising solution to enhance audio quality in wearable devices, particularly when addressing the challenges of reduced sampling rates necessitated by battery life constraints and network instability. However, existing methods either prove computationally prohibitive for mobile platforms, or lack sufficient performance. We present TraMSR, a novel hybrid model combining transformer and Mamba architectures for acoustic speech super-resolution. TraMSR achieves superior performance while significantly reducing computational demands compared to state-of-the-art methods. Our model outperforms GAN-based approaches by up to 7.3% in Perceptual Evaluation of Speech Quality (PESQ) and 1.8% in Short-Time Objective Intelligibility (STOI), with an order of magnitude smaller memory footprint and up to 465 times faster inference speed.

Foundation models excel in tasks such as content generation, zero-shot classifications, and reasoning. However, they struggle with sensing, interacting, and actuating in the physical world due to their dependence on limited sensors and actuators in providing timely contextual information or physical interactions. This reliance restricts the system's adaptability and coverage. To address these issues and create an embodied AI with foundation models (FMs), we introduce Embodied Reconfigurable Drone Agent (EmbodiedRDA). EmbodiedRDA features a custom drone platform that can autonomously swap payloads to reconfigure itself with a diverse list of sensors and actuators. We designed FM agents to instruct the drone to equip itself with appropriate physical modules, analyze sensor data, make decisions, and control the drone's actions. This enables the system to perform a variety of tasks in dynamic physical environments, bridging the gap between the digital and physical worlds.

Kangaroo Mother Care (KMC), involving chest-to-chest skin contact between an infant and caregiver, is proven to be an effective intervention for preterm and full-term infants. Accurate monitoring of KMC duration and infant's vital signs during KMC is clinically important. Existing monitoring methods, however, rely on manual efforts and require rigid sensors or wires/electrodes on the infant's body. We propose Joey, a fabric-based approach to continuously monitor KMC duration and two vital signs essential to an infant's well-being: heart rate and respiration rate. Joey is a soft fabric necklace worn by the caregiver. It leverages the transmission of electrocardiogram (ECG) signals across individuals during skin-to-skin contact. With a minimalist fabric sensor structure, Joey measures KMC duration via the presence of mixed ECG signals. It then isolates the infant's ECG from this mixture with a proposed signal extraction algorithm and employs a diffusion-based denoising model to mitigate motion artifacts, enabling reliable inference of infant's vital signs. We fabricate Joey prototypes with off-the-shelf hardware and evaluate its performance with user studies. Results demonstrate that Joey achieves an average F1 score of 96% for KMC duration measurement, and clinically-acceptable accuracy in infant's vital sign estimation with a mean absolute error of 2.3 beats per minute and 2.9 breaths per minute in estimating heart rate and respiration rate. Clinical interviews further confirm the usability of Joey's sensing fabric for infant skin. A demonstration video of Joey is available at: mobilex.cs.columbia.edu/joey

Kangaroo Mother Care (KMC), involving chest-to-chest skin contact between an infant and caregiver, is proven to be an effective intervention for preterm and full-term infants. Accurate monitoring of KMC duration and infant's vital signs during KMC is clinically important. Existing monitoring methods, however, rely on manual efforts and require rigid sensors or wires/electrodes on the infant's body. We propose Joey, a fabric-based approach to continuously monitor KMC duration and two vital signs essential to an infant's well-being: heart rate and respiration rate. Joey is a soft fabric necklace worn by the caregiver. It leverages the transmission of electrocardiogram (ECG) signals across individuals during skin-to-skin contact. With a minimalist fabric sensor structure, Joey measures KMC duration via the presence of mixed ECG signals. It then isolates the infant's ECG from this mixture with a proposed signal extraction algorithm and employs a diffusion-based denoising model to mitigate motion artifacts, enabling reliable inference of the infant's vital signs. We demonstrate Joey's sensing capability with hand-shaking experiments, showing the real-time mixed ECGs. A demonstration video of Joey for actual KMC practice is available at: mobilex.cs.columbia.edu/joey

The stethoscope is one of the most important diagnostic tools used by healthcare professionals, through a process called auscultation, to screen patients for abnormalities of the heart and lungs. While there are digital stethoscopes on the market which ease this process, it still takes years of training to properly use these devices to listen for abnormal sounds within the body. We present ARSteth, an intelligent stethoscope platform that improves the accessibility of stethoscopes for the general population, allowing anyone to perform auscultation in the comfort of their own homes. Our platform utilizes a combination of augmented reality (AR), acoustic intelligence, and human-machine interaction to dynamically guide users on where to place the stethoscope on different parts of the body (auscultation points), through visual and audio cues. Through user studies, we show that ARSteth, on average, can guide users within 13.2 mm from optimal auscultation points marked by licensed physicians in 13.09 seconds for each auscultation point. By guiding users towards more effective auscultation points, make preventative health screening more accessible and effective for everyone we are able to achieve higher confidence on classifying heart murmurs.

We present a demo showcasing the capabilities of an infrared (IR)-based light communication system with a movable receiver. The system employs IR laser (VCSEL) together with scattering lens as the transmitter and an avalanche photo-diode (APD) with collimator as the receiver, using the reflection cross section existing in the environment (ceilings, walls, etc.) to spread the coverage of the communication system. The demonstration involves transmitting messages encoded as binary data through modulated IR light signals to the movable receiver. The receiver captures the signals using the APD, which are then decoded to retrieve the original message. The demonstration aims to showcase the robustness of the system against various sources of interference and the flexibility of the movable receiver to capture signals from different angles and positions. We believe that our demonstration will be useful for showcasing the potential of IR-based light communication in various applications (e.g., wireless VR/AR and high speed reliable data link) over more traditional communication methods that have limitations such as privacy and bandwidth.

Mapping 3D airflow fields is important for many HVAC, industrial, medical, and home applications. However, current approaches are expensive and time-consuming. We present Anemoi, a sub-$100 drone-based system for autonomously mapping 3D airflow fields in indoor environments. Anemoi leverages the effects of airflow on motor control signals to estimate the magnitude and direction of wind at any given point in space. We introduce an exploration algorithm for selecting optimal waypoints that minimize overall airflow estimation uncertainty. We demonstrate through microbenchmarks and real deployments that Anemoi is able to estimate wind speed and direction with errors up to 0.41 m/s and 25.1° lower than the existing state of the art and map 3D airflow fields with an average RMS error of 0.73 m/s.

We present SoFIT, an easily-deployed and privacy-preserving camera network system for occupant tracking. Unlike traditional camera network-based systems, SoFIT does not require a person to calibrate the network or provide real-world references. This enables anyone, including non-professionals, to install SoFIT. Once installed, SoFIT automatically localizes cameras within the network and generates the floor map leveraging movements of people using the space in daily life, before using the floor map and camera locations to track occupants throughout the environment. We demonstrate through a series of deployments that SoFIT can localize cameras with less than 4.8cm error, generate floor maps with 85% similarity to actual floor maps, and track occupants with less than 7.8cm error.

With the recent societal impact of COVID-19, companies and government agencies alike have turned to thermal camera based skin temperature sensing technology to help screen for fever. However, the cost and deployment restrictions limit the wide use of these thermal sensing technologies. In this work, we present SIFTER, a low-cost system based on a RGB-thermal camera for continuous fever screening of multiple people. This system detects and tracks heads in the RGB and thermal domains and constructs thermal heat map models for each tracked person, and classifies people as having or not having fever. SIFTER can obtain key temperature features of heads in-situ at a distance and produce fever screening predictions in real-time, significantly improving screening through-put while minimizing disruption to normal activities. In our clinic deployment, SIFTER measurement error is within 0.4°F at 2 meters and around 0.6°F at 3.5 meters. In comparison, most infrared thermal scanners on the market costing several thousand dollars have around 1°F measurement error measured within 0.5 meters. SIFTER can achieve 100% true positive rate with 22.5% false positive rate without requiring any human interaction, greatly outperforming our baseline [1], which sees a false positive rate of 78.5%.

Audio is valuable in many mobile, embedded, and cyber-physical systems. We propose AvA, an acoustic adaptive filtering architecture, configurable to a wide range of applications and systems. By incorporating AvA into their own systems, developers can select which sounds to enhance or filter out depending on their application needs. AvA accomplishes this by using a novel adaptive beamforming algorithm called content-informed adaptive beam-forming (CIBF), that directly uses detectors and sound models that developers have created for their own applications to enhance or filter out sounds. CIBF uses a novel three step approach to prop-agate gradients from a wide range of different model types and signal feature representations to learn filter coefficients. We apply AvA to four scenarios and demonstrate that AvA enhances their respective performances by up to 11.1%. We also integrate AvA into two different mobile/embedded platforms with widely different resource constraints and target sounds/noises to show the boosts in performance and robustness these applications can see using AvA.

With the global spread of the COVID-19 pandemic, ventilation indoors is becoming increasingly important in preventing the spread of airborne viruses. However, while sensors exist to measure wind speed and airflow gradients, they must be manually held by a human or an autonomous vehicle, robot, or drone that moves around the space to build an airflow map of the environment. In this demonstration, we present DAE, a novel drone-based system that can automatically navigate and estimate air flow in a space without the need of additional sensors attached onto the drone. DAE directly utilizes the flight controller data that all drones use to self-stabilize in the air to estimate airflow. DAE estimates airflow gradients in a room based on how the flight controller adjusts the motors on the drone to compensate external perturbations and air currents, without the need for attaching additional wind or airflow sensors.

There has been an immense growth in sensors, actuators, and smart devices in recent years, which enable us to better sense, actuate, and understand the physical world. Despite this growth, we have yet to achieve fully intelligent environments. This is, in part, due to the large number of different organizations creating smart devices with proprietary technologies and communication protocols that are not compatible with each other and require significant engineering to incorporate and adapt to specific applications. In this work, we present an easy-to-install and low-cost embedded platform that allows users to rapidly configure a mixture of sensors and actuators. The system is based on the commonly-used Raspberry Pi ecosystem, easily configurable, and does not require users to have prior knowledge of programming, which allows anyone, regardless of background, to use. We also introduce a battery-powered wireless extension module that is suitable for mobile drone applications, where a chord-powered Raspberry Pi is not suitable. We demonstrate the impact our system has on enabling drones with flexible sensing modalities and creating smarter environments by integrating our platform into a variety of intelligent home applications.

Cardiopulmonary ailments are a major cause of mortality. Stethoscopes are one of the most important tools that healthcare professionals use to screen patients for a variety of ailments, especially those related to the heart and lungs. Despite the growth of digital stethoscopes on the market, it takes years of training to properly use stethoscopes to listen for abnormal sounds within the body. In this demonstration, we present an intelligent stethoscope platform that makes stethoscopes more accessible to the general population. Our platform utilizes augmented reality (AR) to provide real-time guidance on where to properly place the stethoscope on the body, enabling the general population to screen themselves for ailments.

In this demonstration, in collaboration with licensed therapists, we introduce an AI therapist that takes advantage of the smart-home environment to screen day-to-day functioning and infer mental wellness of an occupant. Our system can assess a user's daily functioning and mental wellness based on a combination of direct conversation with users and information obtained from smart home devices using psychological rubrics proposed in [1]. We demonstrate that our system can converse with a user in a natural way (through a smartphone or smart speaker) and analyze a user's response semantically and sentimentally. In addition, we show that our system can provide preliminary interventions to help improve the user's wellness. In particular, when abnormal behavior is detected during the conversation or by smart home devices, the system provides psychotherapeutic consolations during the conversation and will check on the occupant's condition by actuating a home robot.

Recent years have witnessed the increasing penetration of wireless charging base stations in the workplace and public areas, such as airports and cafeterias. Such an emerging wireless charging infrastructure has presented opportunities for new indoor localization and identification services for mobile users. In this paper, we present QID, the first system that can identify a Qi-compliant mobile device during wireless charging in real-time. QID extracts features from the clock oscillator and control scheme of the power receiver and employs light-weight algorithms to classify the device. QID adopts a 2-dimensional motion unit to emulate a variety of multi-coil designs of Qi, which allows for fine-grained device fingerprinting. Our results show that QID achieves high recognition accuracy. With the prevalence of public wireless charging stations, our results also have important implications for mobile user privacy.

In this poster abstract, we present a thermal comfort estimation system using low-cost thermal camera based sensor nodes. This system extracts perspective invariant, non-intrusive thermal measurements, is easily deployable and low-cost, and can incorporate individual thermal feedback for more personalized thermal comfort estimates. In comparison with baseline methods, our system is able to improve thermal comfort estimates on the ASHRAE 7-point thermal sensation scale by up to 64% over baseline methods.

Sound detection and classification are critical in many acoustic-based applications. Existing works generally focus on discovering new features and classifiers to improve detection. However, in many scenarios the presence of other sounds may hinder the performance of these sound classifiers. In this work, we take a sound filtering and enhancement approach to improve sound detection for mobile and embedded applications, regardless of the type of detector used.

Vehicle accidents are one of the greatest cause of death and injury in urban areas for pedestrians, workers, and police alike. In this work, we present CSafe, a low power audio-wearable platform that detects, localizes, and provides alerts about oncoming vehicles to improve construction worker safety. Construction worker safety is a much more challenging problem than general urban or pedestrian safety in that the sound of construction tools can be up to orders of magnitude greater than that of vehicles, making vehicle detection and localization exceptionally difficult. To overcome these challenges, we develop a novel sound source separation algorithm, called Probabilistic Template Matching (PTM), as well as a novel noise filtering architecture to remove loud construction noises from our observed signals. We show that our architecture can improve vehicle detection by up to 12% over other state-of-art source separation algorithms. We integrate PTM and our noise filtering architecture into CSafe and show through a series of real-world experiments that CSafe can achieve up to an 82% vehicle detection rate and a 6.90° mean localization error in acoustically noisy construction site scenarios, which is 16% higher and almost 30° lower than the state-of-art audio wearable safety works.

Homes are becoming more intelligent due to the growth of smart sensors and devices found in typical homes. However, most of these sensors and devices function independently from one another, limiting the amount of utility and services a truly "smart" home can provide. In this demonstration, we introduce two key ideas towards more intelligent homes. First, we explore the usage of mobile drones in the home environment. Second, we propose DIA, a system that seamlessly connects to the home environment and automatically discovers and jointly utilizes smart sensors and actuators around the home to provide services that are otherwise not possible. We demonstrate three services that DIA enables.

Energy footprinting has the potential to raise awareness of energy consumption and lead to energy-saving behavior. However, current methods are largely restricted to single buildings; these methods require energy and occupancy monitoring sensor deployments, which can be expensive and difficult to deploy at scale. Further, current methods for estimating energy consumption and population at scale cannot provide fine enough temporal or spatial granularity for a reasonable personal energy footprint estimate. In this work, we present a data-driven system for city-wide estimation of personal energy footprints. This system takes advantage of existing sensing infrastructure and data sources in urban cities to provide energy and population estimates at the building level, even in built environments that do not have existing or accessible energy or population data.

Facial expression monitoring is crucial in fields including mental health care, driver assistant systems, and advertising. However, existing systems typically rely on cameras that capture entire faces, or contact-based bio-signal sensors, which are neither comfortable nor portable. In this demonstration, we present a wireless glasses system for non-contact facial expression monitoring. The system is composed of an IR camera and an embedded processing unit mounted on a 3D-printed glasses frame, and a novel data processing pipeline running across the glasses platform and a computer. Our system performs high-accuracy and real-time facial expression detection with a running time of up to 9 hours. We will show the fully-functioning wearable system in this demonstration.

With the recent societal impact of COVID-19, businesses and government agencies have turned to thermal camera based skin temperature sensing technology to help detect affected civilians. However, cost and deployment restrictions limit the widespread use of these thermal sensing technologies. In this work, we present a low cost system based on an RGB-thermal camera for continuously detecting and estimating facial temperature features for multiple people. This system detects and tracks heads in the RGB and thermal domains, constructs temperature models of individual facial features, and models environmental and surface effects on thermal sensing to reduce temperature measurement error.

In this article, we present recEnergy, a recommender system for reducing energy consumption in commercial buildings with human-in-the-loop. We formulate the building energy optimization problem as a Markov decision process, show how deep reinforcement learning can be used to learn energy-saving recommendations, and effectively engage occupants in energy-saving actions. recEnergy is a recommender system that learns actions with high-energy-saving potential, actively distributes recommendations to occupants in a commercial building, and utilizes feedback from the occupants to learn better energy-saving recommendations. Over a four-week user study, four different types of energy-saving recommendations were trained and learned. recEnergy improves building energy reduction from a baseline saving (passive-only strategy) of 19%-26%.

With the prevalence of smartphones, pedestrians and joggers today often walk or run while listening to music. Since they are deprived of their auditory senses that would have provided important cues to dangers, they are at a much greater risk of being hit by cars or other vehicles. In this paper, we build a wearable system that uses multichannel audio sensors embedded in a headset to help detect and locate cars from their honks, engine, and tire noises, and warn pedestrians of imminent dangers of approaching cars. We demonstrate that using a segmented architecture consisting of headset-mounted audio sensors, a front-end hardware platform that performs signal processing and feature extraction, and machine learning-based classification on a smartphone, we are able to provide early danger detection in real time, from up to 60 m away, and alert the user with low latency and high accuracy. To further reduce power consumption of the battery-powered wearable headset, we implement a custom-designed integrated circuit that is able to compute delays between multiple channels of audio with nW power consumption. A regression-based method for sound source localization, angle via polygonal regression, is proposed and used in combination with the IC to improve the granularity and robustness of localization.

Commercial buildings consume a large portion of the total electricity in the United States. One method for energy saving in commercial buildings targets inefficiencies of unoccupied spaces by relaxing the setpoint temperature. However, energy savings are severely limited when occupants are assumed to be "immovable objects"; instead, by encouraging occupant participation in the optimization, a much greater amount of energy savings can be achieved. In this work, we build on this idea and introduce energy saving recommendations based on occupant location. We introduce two types of energy saving recommendations based on location: move recommendations, which recommends the occupant to move from one space to another, and shift schedule recommendations, which recommends the occupant to arrive or depart a set amount of time earlier or later. To investigate the effects of the energy saving recommendations, we introduced a tightly coupled system composing of a simulator and a recommender system. Simulations in our building testbed revealed that energy saving recommendations coupled with occupancy-based HVAC energy management saves 25% more energy than occupancy-based HVAC energy management alone.

No abstract available.

We propose a system that tracks each occupant’s personal share of energy use, or “energy footprint,” inside commercial building environments and provides insights to occupants on the real-time energy impact of their actions. We propose a new space-centric policy for fair apportionment of energy in shared environments and demonstrate a method for automatically determining space-centric energy zones. In this work, we design and implement ePrints, a system for tracking personalized energy usage in real-time. ePrints supports different apportionment policies, with microsecond-level footprint computation time and graceful scaling with size of building, frequency of energy updates, and rate of occupant location changes. Finally, we present applications enabled by our system, such as mobile and wearable applications to provide users timely feedback on the energy impacts of their actions, as well as applications to provide energy saving suggestions and inform building-level policies.

With smart buildings becoming popular, it is important to track the wastage of energy in public shared buildings to save energy. Current monitoring systems do not provide real-time visibility into the impact of occupants' actions on energy consumption of a building. We propose a system that tracks the energy consumed by users in shared spaces such as offices thereby making them aware and accountable for the energy they consume. Our system combines energy monitoring with localization techniques to generate real-time energy footprints for every occupant in a shared space and provides actionable feedback to them in the form of visualization.

With the prevalence of smartphones, pedestrians and joggers today often walk or run while listening to music. Since they are deprived of their auditory senses that would have provided important cues to dangers, they are at a much greater risk of being hit by cars or other vehicles. In this demonstration we present SEUS, a wearable system aimed at Sense Enhancement for Urban Safety. SEUS uses a three-stage architecture, consisting of headset mounted audio sensors, an embedded front-end for signal processing and feature extraction, and machine learning based classification on a smartphone, to provide early danger detection for pedestrians in real-time.

We present the design, implementation, and evaluation of AirCloud -- a novel client-cloud system for pervasive and personal air-quality monitoring at low cost. At the frontend, we create two types of Internet-connected particulate matter (PM2:5) monitors -- AQM and miniAQM, with carefully designed mechanical structures for optimal air-flow. On the cloud-side, we create an air-quality analytics engine that learn and create models of air-quality based on a fusion of sensor data. This engine is used to calibrate AQMs and mini-AQMs in real-time, and infer PM2:5 concentrations. We evaluate AirCloud using 5 months of data and 2 month of continuous deployment, and show that AirCloud is able to achieve good accuracies at much lower cost than previous solutions. We also show three real applications built on top of AirCloud by 3rd party developers to further demonstrate the value of our system.

We present the design, implementation, and preliminary results of PAM - a low-cost and portable personal air quality monitor that provides real-time air quality measurements for the user's immediate environment. PAM consists of a PAM client and a cloud-side PAM service. The PAM client utilizes two inexpensive dust sensors to obtain raw particulate concentration measurement, and two Arduino boards for local filtering, web services, and communication with PAM service. The cloud-side PAM service constructs a statistical model of air quality and provides PAM clients with context-dependent calibration curves. Together, PAM is able to approximate PM2.5, PM10, and AQI measurement for users in-situ and at low cost.

Auditeur is a general-purpose, energy-efficient, and context-aware acoustic event detection platform for smartphones. It enables app developers to have their app register for and get notified on a wide variety of acoustic events. Auditeur is backed by a cloud service to store user contributed sound clips and to generate an energy-efficient and context-aware classification plan for the phone. When an acoustic event type has been registered, the smartphone instantiates the necessary acoustic processing modules and wires them together to execute the plan. The phone then captures, processes, and classifies acoustic events locally and efficiently. Our analysis on user-contributed empirical data shows that Auditeur's energy-aware acoustic feature selection algorithm is capable of increasing the device lifetime by 33.4%, sacrificing less than 2% of the maximum achievable accuracy. We implement seven apps with Auditeur, and deploy them in real-world scenarios to demonstrate that Auditeur is versatile, 11.04% - 441.42% less power hungry, and 10.71% - 13.86% more accurate in detecting acoustic events, compared to state-of-the-art techniques. We present a user study to demonstrate that novice programmers can implement the core logic of interesting apps with Auditeur in less than 30 minutes, using only 15 - 20 lines of Java code.

A mobile phone, as a pervasive device, has great potential in human wellness monitoring. In this demo, we first present the design and implementation of our hardware - SEPTIMU. SEPTIMU consists of a small baseboard and a pair of tiny sensor boards embedded inside conventional earphones. The baseboard provides power conversion and data communication through the normal audio jack interface. The embedded sensor board is 1×1cm2 and integrates 3-axis accelerometer, gyroscope, thermometer, photodiode and microphone. Secondly, we evaluate SEPTIMU using a mobile application that continuously monitors body posture and provides feedback to the user.

Many indoor sensing applications leverage knowledge of relative proximity among physical objects and humans, such as the notion of "within arm's reach". In this paper, we quantify this notion using "proximity zone", and propose a methodology that empirically and systematically compare the proximity zones created by various wireless technologies. We find that existing technologies such as 802.15.4, Bluetooth Low Energy (BLE), and RFID fall short on metrics such as boundary sharpness, robustness against interference, and obstacle penetration. We then present the design and evaluation of a wireless proximity detection platform based on magnetic induction - LiveSynergy. LiveSynergy provides sweet spot for indoor applications that require reliable and precise proximity detection. Finally, we present the design and evaluation of an end-to-end system, deployed inside a large food court to offer context-aware and personalized advertisements and diet suggestions at a per-counter granularity.

It is a tedious task to search and locate a specific book from massive number of books arbitrarily placed in a bookshelf. In this paper, we demonstrate iBookshelf, a system which allows users to quickly search and accurately locate books in the bookshelf, by leveraging a passive RFID system. By deploying a number of reference tags on the bookshelf, we are able to perform localization based on the similarities in received signal strength, and effectively offset the impact from the ambient noises and interferences. We deploy and evaluate our system in a real 3m x 2.5m bookshelf, and show that users are able to locate the book from our Android-based application with 85% accuracy.

Mobile phones have become an ideal platform for physiological and environmental sensing. A number of research and commercial smartphone "accessories" have emerged in recent years that try to extend the sensing capabilities of a mobile phone. However, the major drawback of these devices is that they either require the user to act in some specific way or change their lifestyle and habit to some extent. In this demo, we present Septimu V2 (Septimu2) -- a novel non-intrusive physiological and environmental sensing platform which is fully embedded in a conventional earphone, works with existing smartphones, and does not require the user to change habits in any way. Septimu2 is a continuation of [1], and integrates a suite of new sensors. In addition to 3-axis accelerometer and gyroscope, Septimu2 incorporates remote IR temperature sensor, IR LED, IR photodiode and two additional microphones. The baseboard performs signal condition and sends the data to cellphone via Bluetooth. Septimu2 enables a number of applications, including heart-rate monitoring, fine grained posture detection, and external sound source localization and classification.

MusicalHeart is a biofeedback-based, context-aware, automated music recommendation system for smartphones. We introduce a new wearable sensing platform, Septimu, which consists of a pair of sensor-equipped earphones that communicate to the smartphone via the audio jack. The Septimu platform enables the MusicalHeart application to continuously monitor the heart rate and activity level of the user while listening to music. The physiological information and contextual information are then sent to a remote server, which provides dynamic music suggestions to help the user maintain a target heart rate. We provide empirical evidence that the measured heart rate is 75% -- 85% correlated to the ground truth with an average error of 7.5 BPM. The accuracy of the person-specific, 3-class activity level detector is on average 96.8%, where these activity levels are separated based on their differing impacts on heart rate. We demonstrate the practicality of MusicalHeart by deploying it in two real world scenarios and show that MusicalHeart helps the user achieve a desired heart rate intensity with an average error of less than 12.2%, and its quality of recommendation improves over time.

Smart phones provide a convenient platform for social applications with their rich set of sensors and communication interfaces, and have become an integral part of our daily lives. In Genius-on-the-Go, we hope to explore a new social sharing mechanism for mobile users based on proximity. Our key contribution is an efficient discovery layer, and enabling interactions with nearby devices using FM radio. But since current mobile phones do not export radio drivers that support transmit mode (even though most hardware chips are already capable of), we designed a custom accessory incorporating an FM chip and communicating with the phone via the headphone jack. Using this accessory, we demonstrate efficient discovery and communication between nearby smart phones, and show an mobile-DJ application built on top.

In this demonstration, we present the architecture, implementation, and applications of LiveSynergy --- a system that provides reliable proximity sensing and open interactive abstractions for physical spaces and objects, to enable rich interactions between humans and their environment.

We present the architecture, specification, and implementations of a simple monitoring and action profile (sMAP), optimized for sensors, meters, and actuators in building environments. Our architecture is built on HTTP/REST and uses JSON as the object format for interoperability. We implement sMAP on a variety of resource monitors and actuators inside a commercial building, including mote-based wireless sensors and meters running IPv6/6LowPAN, Modbus based panel meters, and external data sources. We show that sMAP is widely implementable and efficient, and our API and schema definitions are expressive and concise. We demonstrate that our architecture is well suited for resource constrained devices using compressed JSON and proxies.

As more and more physical information becomes available, a critical problem is enabling the simple and efficient exchange of this data. We present our design for a simple RESTful web service called the Simple Measuring and Actuation Profile (sMAP) which allows instruments and other producers of physical information to directly publish their data. In our design study, we consider what information should be represented, and how it fits into the RESTful paradigm. To evaluate sMAP, we implement a large number of data sources using this profile, and consider how easy it is to use to build new applications. We also design and evaluate a set of adaptations made at each layer of the protocol stack which allow sMAP to run on constrained devices.

We describe the design, deployment, and experience with a wireless sensor network for high-fidelity monitoring of electrical usage in buildings. A network of 38 mote-class AC meters, 6 light sensors, and 1 vibration sensor is used to determine and audit the energy envelope of an active laboratory. Classic WSN issues of coverage, aggregation, sampling, and inference are shown to appear in a novel form in this context. The fundamental structuring principle is the underlying load tree, and a variety of techniques are described to disambiguate loads within this structure. Utilizing contextual metadata, this information is recomposed in terms of its spatial, functional, and individual projections. This suggests a path to broad use of WSN technology in energy and environmental domains.

We present a building block approach to hardware platform design based on a decade of collective experience in this area, arriving at an architecture in which general-purpose modules that require expertise to de sign and incorporate commonly-used functionality are integrated with application-specific carriers that satisfy the unique sensing, power supply, and mechanical constraints of an application. Of course, modules are widespread, but our focus is far less on the performance of any individual module and far more on an overall architecture that supports the prototype, pilot, and production stages of design, and preserves the artifacts and learnings accumulated along the way.

A home where every major appliance can be monitored for energy consumption and individually controlled wirelessly has long been a dream of gadgeteers and the green-conscious alike. Research has shown that real-time, per-appliance electricity usage feedback can induce behavior changes that lead to 10% to 20% reduction in usage [2].

We present SPOT, a scalable power observation tool that enables in situ measurement of nodal power and energy over a dynamic range exceeding four decades or a temporal resolution of microseconds. Using SPOT, every node in a sensor network can now be instrumented, providing unparalleled visibility into the dynamic power profile of applications and system software. Power metering at every node enables previously impossible empirical evaluation of low power designs at scale. The SPOT architecture and design meet challenges unique to wireless sensor networks and other low power systems, such as orders of magnitude difference in current draws between sleep and active states, short-duration power spikes during periods of brief activity, and the need for minimum perturbation of the system under observation.

Environmental energy is an attractive power source for low power wireless sensor networks. We present Prometheus, a system that intelligently manages energy transfer for perpetual operation without human intervention or servicing. Combining positive attributes of different energy storage elements and leveraging the intelligence of the microprocessor, we introduce an efficient multi-stage energy transfer system that reduces the common limitations of single energy storage systems to achieve near perpetual operation. We present our design choices, tradeoffs, circuit evaluations, performance analysis, and models. We discuss the relationships between system components and identify optimal hardware choices to meet an application's needs. Finally we present our implementation of a real system that uses solar energy to power Berkeley's Telos Mote. Our analysis predicts the system will operate for 43 years under 1% load, 4 years under 10% load, and 1 year under 100% load. Our implementation uses a two stage storage system consisting of supercapacitors (primary buffer) and a lithium rechargeable battery (secondary buffer). The mote has full knowledge of power levels and intelligently manages energy transfer to maximize lifetime.

This paper presents a study of how empirical ranging characteristics affect multihop localization in wireless sensor networks. We use an objective metric to evaluate a well-established parametric model of ranging called Noisy Disk: if the model accurately predicts the results of a real-world deployment, it sufficiently captures ranging characteristics. When the model does not predict accurately, we systematically replace components of the model with empirical ranging characteristics to identify which components contribute to the discrepancy. We reveal that both the connectivity and noise components of Noisy Disk fail to accurately represent real-world ranging characteristics and show that these shortcomings affect localization in different ways under different circumstances.