A confluence of advances in several fields has led to the emergence of mixed-reality headsets. They can enable us to interact with and visualize our environments in novel ways. Nonetheless, mixed-reality headsets are constrained today as their camera systems only capture a narrow part of the visible spectrum. Our environment contains rich information that cameras do not capture. It includes phenomena captured through sensors, electromagnetic fields beyond visible light, acoustic emissions, and magnetic fields. We demonstrate our ongoing work, PixelGen, to redesign cameras for low power consumption and to be able to visualize our environments in a novel manner, making some of the invisible phenomena visible. Pixel-Gen combines low-bandwidth sensors with a monochrome camera to capture a rich representation of the world. This design choice ensures information is communicated energy-efficiently. This information is then combined with diffusion-based image models to generate unique representations of the environment, visualizing the otherwise invisible fields. We demonstrate that together with a mixed reality headset, it enables us to observe the world uniquely.

Mixed-reality headsets offer new ways to perceive our environment. They employ visible spectrum cameras to capture and display the environment on screens in front of the user's eyes. However, these cameras lead to limitations. Firstly, they capture only a partial view of the environment. They are positioned to capture whatever is in front of the user, thus creating blind spots during complete immersion and failing to detect events outside the restricted field of view. Secondly, they capture only visible light fields, ignoring other fields like acoustics and radio that are also present in the environment. Finally, these power-hungry cameras rapidly deplete the mixed-reality headset's battery. We introduce PixelGen to rethink embedded cameras for mixed-reality headsets. PixelGen proposes to decouple cameras from the mixed-reality headset and balance resolution and fidelity to minimize the power consumption. It employs low-resolution, monochrome image sensors and environmental sensors to capture the surroundings around the headset. This approach reduces the system's communication bandwidth and power consumption. A transformer-based language and image model process this information to overcome resolution trade-offs, thus generating a higher-resolution representation of the environment. We present initial experiments that show PixelGen's viability.

Wireless embedded systems assist us in collecting data from the physical world, through sensor data analysis, such systems allow us to understand our environment. However, deploying wireless embedded systems and analyzing the collected data remains significantly challenging. This is due to the steep learning curve required to implement custom machine-learning models and other algorithms for data analysis. Furthermore, it is also challenging to program individual embedded devices. The diversity of the available platforms and their capabilities further compounds this problem. In response, we introduce an end-to-end system, called the Otter. It facilitates simple sensor data collection using commodity-embedded platforms. Moreover, it employs a large language model to design a natural language interface for the analysis and extraction of useful information from the sensor data. We present our preliminary work on prototyping this system, applying it to a specific use case of hand gesture detection. Otter represents one of the first systems to leverage the enhanced capabilities of large language models for simplifying wireless embedded system deployments.

An important step in the deployment of wireless embedded systems is the analysis of the sensor data. Traditionally, this requires machine learning models tailored to the application use case. However, this step requires significant expertise from the end user and can be less adaptable to the dynamics of real-world deployments. In recent years, large language models have seen significant developments. These models have been shown to be capable of performing general-purpose tasks. In this work, we explore the hypothesis that large language models can be used to aid in sensor data analysis. Our preliminary findings through real-world experiments show significant promise for two tasks: inferring hand gestures through tracking of light and vibration sensor data. We believe these findings highlight the potential of large language models in sensor data analysis, and thus, it warrants further study.

The rapid growth in wireless embedded systems is threatened by the challenges associated with programming and deploying them. In addition, there is also the complexity inherent in analyzing of the sensor data. Notably, these tasks require high levels of end-user expertise. In this way, an entry barrier is introduced to deploying wireless embedded systems. In this work, we introduce Otter, an end-to-end system designed to simplify these tasks by leveraging the emergent properties of large language models. We demonstrate that Otter allows commodity embedded platforms to capture sensor data, such as light and vibration sensors, which can then be used to identify hand gestures in a near real-time manner. This is all while being prompted using natural language prompts by the end-user. Otter is the first system of its kind and has the potential to facilitate wireless embedded systems proliferation significantly.