While DNN models have dramatically improved the accuracy of machine vision tasks, pervasive deployments of vision sensors for surveillance tasks continue to suffer from high energy consumption and network traffic overhead. To tackle these problems, we introduce CollabCam , an edge-based machine vision system designed for multi-camera deployments that leverages the naturally-occurring overlaps in the field-of-view (FoV) among neighboring cameras. CollabCam synergistically combines two innovative ideas: (a) having each individual camera compose and transmit mixed-resolution frames (MRF) via lightweight down-sampling, where the transmitted images have significantly lower resolution in the shared, overlapping portions , and (b) performing inference, for an exemplar object detection task, for each camera stream using a new collaborative mechanism which utilizes suitably-translated object bounding boxes from a peer “collaborating” camera as an additional input channel. We demonstrate how this collaborative mechanism is generalizable and can be realized by simply retraining off-the-shelf object detector DNNs, such as YOLOv3 and SSD, without modifying their model structures. By emulating the performance of CollabCam using two benchmark outdoor-campus multi-camera datasets, we show that Collab-DNNs can accommodate a 50–60 fold reduction in image size (therefore reducing network transmission overhead), for both high-resolution (1056x1056) and low-resolution (512x512) images, with a modest ≤ 2 - 5% drop in object detection accuracy, compared to a non-collaborative approach that suffers a ∼ 45–60% drop in accuracy. Subsequently, by deploying a Raspberry-Pi based CollabCam prototype on a campus-based test-bed, we demonstrate that CollabCam can reduce the overall energy/image frame overhead by ∼25–35%, with even higher energy savings (∼35–45%) likely with hardware optimization. Finally, additional experiments help demonstrate that CollabCam can prove beneficial for varied (including multi-class) object detection tasks and that CollabCam’s performance benefits may be best realized by ensuring that the number of deployed, collaborating cameras is not excessively high.

Rash driving detection is vital to prevent accidents and improve public safety. Existing rash driving solutions using hand-crafted features have several limitations. We propose a simple yet efficient two-step process to overcome the limitations of the existing works by leveraging smartphone sensor (accelerometer and gyroscope) data. The first step filters out normal driving data and retains only the abnormal driving data with the proposed Adaptive Time Window (ATW) algorithm. This not only enhances the accuracy of detection but also reduces computation time, making our solution more efficient. Importantly, the proposed ATW algorithm completely eliminates window overlap redundancy and edge effects in the system. The second step classifies abnormal driving patterns with the proposed 1D Convolutional Neural Network (CNN) model. Our results demonstrate that the proposed solution is highly accurate and has a weighted accuracy of 97.14%. Additionally, as part of this research, we have curated and released a labeled Indian dataset comprising five distinct rash driving patterns: Lane Weaving, Lane Swerving, Hard Braking, Hard Cornering, and Quick U-turn. This dataset can be valuable for further studies and aid in developing multimodal rash driving detection systems.

Tracking movements in the environment of macroscopic objects enables numerous applications, from monitoring vital signs through body movements to inferring hand gestures. However, current systems overwhelmingly rely on contact-based sensors or energy-consuming radio frequency mechanisms that necessitate complex radio transceivers for receptions. We present ongoing research on a novel low-power sensor that leverages the unique characteristics of tunnel diodes. This sensor can detect minute changes in its vicinity and communicate these changes over radio waves, all while consuming under 150 microwatts of power consumption. Notably, the transmitted radio waves are processed using low-cost, off-the-shelf radio transceivers, resulting in low cost and power consumption. The sensor’s functionality stems from the sensitivity of the resonant frequency of the tunnel diode oscillators to changes in their electromagnetic surroundings. Our early work exhibits its potential for detecting a person’s breathing patterns, and hand gestures.

A confluence of advances in several fields has led to the emergence of mixed-reality headsets. They can enable us to interact with and visualize our environments in novel ways. Nonetheless, mixed-reality headsets are constrained today as their camera systems only capture a narrow part of the visible spectrum. Our environment contains rich information that cameras do not capture. It includes phenomena captured through sensors, electromagnetic fields beyond visible light, acoustic emissions, and magnetic fields. We demonstrate our ongoing work, PixelGen, to redesign cameras for low power consumption and to be able to visualize our environments in a novel manner, making some of the invisible phenomena visible. Pixel-Gen combines low-bandwidth sensors with a monochrome camera to capture a rich representation of the world. This design choice ensures information is communicated energy-efficiently. This information is then combined with diffusion-based image models to generate unique representations of the environment, visualizing the otherwise invisible fields. We demonstrate that together with a mixed reality headset, it enables us to observe the world uniquely.

Mixed-reality headsets offer new ways to perceive our environment. They employ visible spectrum cameras to capture and display the environment on screens in front of the user's eyes. However, these cameras lead to limitations. Firstly, they capture only a partial view of the environment. They are positioned to capture whatever is in front of the user, thus creating blind spots during complete immersion and failing to detect events outside the restricted field of view. Secondly, they capture only visible light fields, ignoring other fields like acoustics and radio that are also present in the environment. Finally, these power-hungry cameras rapidly deplete the mixed-reality headset's battery. We introduce PixelGen to rethink embedded cameras for mixed-reality headsets. PixelGen proposes to decouple cameras from the mixed-reality headset and balance resolution and fidelity to minimize the power consumption. It employs low-resolution, monochrome image sensors and environmental sensors to capture the surroundings around the headset. This approach reduces the system's communication bandwidth and power consumption. A transformer-based language and image model process this information to overcome resolution trade-offs, thus generating a higher-resolution representation of the environment. We present initial experiments that show PixelGen's viability.

We present the hardware design and software pipeline for an ultra-low power device, in the form factor of a wearable badge, that supports energy efficient sensing, processing and wireless transfer of human voice commands and interactions. The proposed system, called PA2BLO, is envisioned to support both: (a) real-time, scalable, authorized voice based interaction and control of devices and appliances, and (b) longitudinal, low-power logging of natural voice interactions. PA2BLO in-troduces two key novel capabilities. First, it includes a low power, low-complexity voice authentication module that is able to reliably authenticate an authorized user only using low sampling rate (500 Hz) audio data. Second, to reduce concerns around inadvertent leakage of voice biometrics to less secure voice-driven services, PA2BLO uses a power-efficient, randomized pitch shifting technique that dramatically lowers the ability to perform speaker recognition while preserving instruction/speech comprehensibility. We describe PA2BLO's Cortex M4F-based micro-controller based hardware implementation, which is care-fully designed to eliminate redundant processing and consumes less than 50J of energy per hour of active voice capture and processing. Through both controlled and naturalistic studies, we show that the PA2BLO prototype is capable of authenticating user voice segments reliably (accuracy> 89.8%) and can operate for well over a day (using a supercapacitor charged within just one minute) while capturing 2+ hours of active speaker data.

A stark disparity exists in the energy consumption for performing transmissions and the tasks of sensing and processing in wireless embedded systems. We present our early work to design a novel transmitter that enables transmissions at a similar energy consumption as other tasks in wireless embedded systems. In particular, the proposed transmitter does not require a carrier emitting device, which is essential to support backscatter transmitters, but has also limited their widespread deployment. The proposed transmitter exploits the capability of tunnel diode oscillators to function as a low-power, self-oscillating mixer. This property enables us to mix a weak baseband signal with a locally generated carrier signal at a peak power consumption below 100 microwatts. Nevertheless, the tunnel diode oscillator trades off the stability for low energy consumption leading to poor link reliability. In this study, we investigate error correction codes to increase link reliability. Our experiments demonstrate the potential of the transmitter to support short-range transmissions with a low energy consumption and enhanced link reliability.

Wireless embedded systems assist us in collecting data from the physical world, through sensor data analysis, such systems allow us to understand our environment. However, deploying wireless embedded systems and analyzing the collected data remains significantly challenging. This is due to the steep learning curve required to implement custom machine-learning models and other algorithms for data analysis. Furthermore, it is also challenging to program individual embedded devices. The diversity of the available platforms and their capabilities further compounds this problem. In response, we introduce an end-to-end system, called the Otter. It facilitates simple sensor data collection using commodity-embedded platforms. Moreover, it employs a large language model to design a natural language interface for the analysis and extraction of useful information from the sensor data. We present our preliminary work on prototyping this system, applying it to a specific use case of hand gesture detection. Otter represents one of the first systems to leverage the enhanced capabilities of large language models for simplifying wireless embedded system deployments.

The rapid growth in wireless embedded systems is threatened by the challenges associated with programming and deploying them. In addition, there is also the complexity inherent in analyzing of the sensor data. Notably, these tasks require high levels of end-user expertise. In this way, an entry barrier is introduced to deploying wireless embedded systems. In this work, we introduce Otter, an end-to-end system designed to simplify these tasks by leveraging the emergent properties of large language models. We demonstrate that Otter allows commodity embedded platforms to capture sensor data, such as light and vibration sensors, which can then be used to identify hand gestures in a near real-time manner. This is all while being prompted using natural language prompts by the end-user. Otter is the first system of its kind and has the potential to facilitate wireless embedded systems proliferation significantly.