Internet of Things (IoT) provides a wide range of services to improve convenience and comfort in our daily lives. However, various sensors equipped on IoT devices often raise privacy concerns. Prior works on privacy focus on passive protection from the data and device perspective, such as data encryption and communication protocol design. In this work, we introduce PrivaSee, an augmented reality (AR)-enabled privacy visualization platform to empower users with proactive privacy protection by enhancing their understanding of privacy perception for multimodal sensors.

Remote sports coaching connects athletes to interactive training sessions, despite busy schedules and/or lack of access to local trainers. In current formats, athletes record videos of themselves, which they send to coaches for feedback or use in self-coaching. Additionally, some coaches turn to video conferencing platforms such as FaceTime or Zoom for live coaching. A significant challenge with these methods of remote sports coaching is the absence of spatial analysis capabilities, which hinders in-depth assessment of athletic performance.This paper introduces Augmented Coach, an immersive and interactive sports coaching system. Augmented Coach utilizes volumetric data to reconstruct the 3D representations of the athletes. As a result, coaches can not only view the resulting point cloud videos of the athletes performing athletic movements, but also employ the system’s spatial annotation and visualization tools to gain insights into movement patterns and communicate with remote athletes. Unlike existing tools tailored to specific sports, Augmented Coach explores spatial kinesthetic values shared across various sports through a pilot study and designs adaptable features applicable to diverse sports coaching scenarios. To assess the system’s usability, we conducted a user study involving ten users, spanning certified coaches and experienced athletes from various sports, illustrating how they can utilize the system’s features to enhance coaching in their respective disciplines.

Energy-efficient visual sensing is of paramount importance to enable battery-backed low power IoT and mobile applications. Unfortunately, modern image sensors still consume hundreds of milliwatts of power, mainly due to analog readout. This is because current systems always supply a fixed voltage to the sensor's analog circuitry, leading to higher power profiles. In this work, we propose to aggressively scale the analog voltage supplied to the camera as a means to significantly reduce sensor power consumption. To that end, we characterize the power and fidelity implications of analog voltage scaling on three off-the-shelf image sensors. Our characterization reveals that analog voltage scaling reduces sensor power but also degrades image quality. Furthermore, the degradation in image quality situationally affects the task accuracy of vision applications.

Energy-efficient visual sensing is crucial for enabling battery-backed low power IoT and mobile applications. However, modern image sensors still exhibit high power consumption, primarily attributed to analog readout. This high power consumption arises from the conventional practice of supplying a fixed voltage to the analog circuitry of image sensors. Towards that end, we propose to aggressively scale analog voltage supplied to the camera as means to significantly reduce sensor power consumption. This demonstration showcases the potential of dynamic voltage scaling on commercial image sensors within an RPi-based video streaming pipeline. We enable users to flexibly configure sensor voltage settings and observe the impact on both image fidelity and sensor power consumption. Furthermore, this demonstration offers a framework for studying the implications of flexible voltage specification in the context of a person tracking application.

What we feel from handling liquids in vessels produces unmistakably fluid tactile sensations. These stimulate essential perceptions in home, laboratory, or industrial contexts. Feeling fluid interactions from virtual fluids would similarly enrich experiences in virtual reality. We introduce Geppetteau, a novel string-driven weight shifting mechanism capable of providing perceivable tactile sensations of handling virtual liquids within a variety of vessel shapes. These mechanisms widen the range of augmentable shapes beyond the state-of-the-art of existing mechanical systems. In this work, Geppetteau is integrated into conical, spherical, cylindrical, and cuboid shaped vessels. Variations of these shapes are often used for fluid containers in our day-to-day. We studied the effectiveness of Geppetteau in simulating fine and coarse-grained tactile sensations of virtual liquids across three user studies. Participants found Geppetteau successful in providing congruent physical sensations of handling virtual liquids in a variety of physical vessel shapes and virtual liquid volumes and viscosities.

Huge advancements have been made over the years in terms of modern image-sensing hardware and visual computing algorithms (e.g., computer vision, image processing, and computational photography). However, to this day, there still exists a current gap between the hardware and software design in an imaging system, which silos one research domain from another. Bridging this gap is the key to unlocking new visual computing capabilities for end applications in commercial photography, industrial inspection, and robotics. In this survey, we explore existing works in the literature that can be leveraged to replace conventional hardware components in an imaging system with software for enhanced reconfigurability. As a result, the user can program the image sensor in a way best suited to the end application. We refer to this as software-defined imaging (SDI), where image sensor behavior can be altered by the system software depending on the user’s needs. The scope of our survey covers imaging systems for single-image capture, multi-image, and burst photography, as well as video. We review works related to the sensor primitives, image signal processor (ISP) pipeline, computer architecture, and operating system elements of the SDI stack. Finally, we outline the infrastructure and resources for SDI systems, and we also discuss possible future research directions for the field.

Energy-efficient visual sensing is extremely important to enable battery powered mobile and IoT applications. While several scheduling techniques have been proposed to save the digital power of sensing, the analog power [1] of capturing an image still remains a daunting barrier for a camera's energy-efficiency. To that end, we characterize the power and performance implications of analog voltage scaling on off-the-shelf image sensors. Our characterization reveals that while reducing the analog voltage supplied to image sensor helps promote sensor power efficiency, it also impairs imaging fidelity, specifically by making images brighter and noisier. Furthermore, we find that brighter and noisier images situationally affect the task accuracy of vision applications. In this poster, we propose an investigation towards a system that adaptively scales analog voltage to optimize sensor energy, while respecting the fidelity needs of visual tasks.

Remote coaching for sports is challenged by the lack of 3D spatial communication. While athletes send live or recorded videos to their coaches, these 2D representations fail to capture the spatial relationships of the body, limiting the ability to understand timing, weight distribution, and smoothness in an athletic movement. This demonstration presents Augmented Coach, an AR sports coaching platform for coaches to remotely view, manipulate, and annotate athletic movements in 3D augmented space. Also, this demonstration provides an adaptive platform to study real-time efficient volumetric data transmission between remotely connected devices, including over 5G cellular networks.

Mimicking physical odor sensations virtually can present users with a real - time odor synthesis that approximates what users would smell in a virtual environment, e.g., as they walk around in virtual reality. To this end, we devise a Smell Engine that includes: (i) a Smell Composer framework that allows developers to configure odor sources in virtual space, (ii) a Smell Mixer that dynamically estimates the odor mix that the user would smell, based on diffusion models and relative odor source distances, and (iii) a Smell Controller that coordinates an olfactometer to physically present an approximation of the odor mix to the user’s mask from a set of odorants channeled through controllable flow valves. Through a three - part user study, we found that the Smell Engine can help measure a subject’s olfactory detection threshold and improve their ability to precisely localize odors in the virtual environment, as compared to existing trigger - based solutions.

High spatiotemporal resolution can offer high precision for vision applications, which is particularly useful to capture the nuances of visual features, such as for augmented reality. Unfortunately, capturing and processing high spatiotemporal visual frames generates energy-expensive memory traffic. On the other hand, low resolution frames can reduce pixel memory throughput, but reduce also the opportunities of high-precision visual sensing. However, our intuition is that not all parts of the scene need to be captured at a uniform resolution. Selectively and opportunistically reducing resolution for different regions of image frames can yield high-precision visual computing at energy-efficient memory data rates.

Augmented Reality (AR) is becoming readily more available as the number of AR capable smartphones and tablets increase in popularity. With its exponential development, augmented reality offers an oppurtunity to facilitate education in online chemistry. In hopes of furthering the advancement of augmented reality in online chemistry education, we developed a boiling water experiment to show the effects of heat capacity and to create an interactive lab experiment for online learning. Our work-in-progress paper explores how the utilization of augmented reality can improve the learning process and better exhibit chemistry lab concepts.

Augmented Reality (AR) enables smartphone users to interact with virtual content spatially overlaid on a continuously captured physical world. Under the current permission enforcement model in popular operating systems, AR apps are given Internet permission at installation time, and request camera permission and external storage write permission at runtime through a user's approval. With these permissions granted, any Internet-enabled AR app could silently collect camera frames and derived visual information for malicious intent without a user's awareness. This raises serious concerns about the disclosure of private user data in their living environments.

Remote sensing is currently the primary method of obtaining knowledge about the composition and physical properties of the surface of other planets. In a commonly used technique, visible and near-infrared (VNIR) spectrometers onboard orbiting satellites capture reflectance data at different wavelengths, which in turn gives insight about the minerals present and the overall composition of the terrain. In select locations on Mars, rovers have also conducted up close in-situ investigation of the same terrains examined by orbiters, allowing direct comparisons at different spatial scales. In this work, we build Planetary Visor, a virtual reality tool to visualize orbital and ground data around NASA's Mars Science Laboratory Curiosity rover's ongoing traverse in Gale Crater. We have built a 3D terrain along Curiosity's traverse using rover images, and within it we visualize satellite data as polyhedrons, superimposed on that terrain. This system provides perspectives of VNIR spectroscopic data from a satellite aligned with ground images from the rover, allowing the user to explore both the physical aspects of the terrain and their relation to the mineral composition. The result is a system that provides seamless rendering of datasets at vastly different scales. We conduct a user study with subject matter experts to evaluate the success and potential of our tool. The results indicate that Visor assists with geometric understanding of spectral data, improved geological context, a better sense of scale while navigating terrain, and new insights into spectral data. The result is not only an immersive environment in a scientifically interesting area on Mars, but a robust tool for analysis and visualization of data that can yield improved scientific discovery. This technology is relevant to the ongoing operations of the Curiosity rover and will directly be able to represent the data collected in the upcoming Mars 2020 Perseverance rover mission.

With the ever-expanding development of affordable and available virtual reality headsets, there is a great opportunity for improving learning outcomes through virtual learning. Towards a future of online chemistry education, we present a titration experiment with a corresponding physical haptic burette to embed the physical sensations of labwork inside virtual reality learning for broad access. Our work in progress paper investigates how tactile interactions into fully immersive worlds can improve learning outcomes in multi-modal manners and lead to better understanding and retention of key chemistry concepts, e.g., titration principles.

Mobile vision systems would benefit from the ability to situationally sacrifice image resolution to save system energy when imaging detail is unnecessary. Unfortunately, any change in sensor resolution leads to a substantial pause in frame delivery -- as much as 280 ms. Frame delivery is bottlenecked by a sequence of reconfiguration procedures and memory management in current operating systems before it resumes at the new resolution. This latency from reconfiguration impedes the adoption of otherwise beneficial resolution-energy tradeoff mechanisms. We propose Banner as a media framework that provides a rapid sensor resolution reconfiguration service as a modification to common media frameworks, e.g., V4L2. Banner completely eliminates the frame-to-frame reconfiguration latency (226 ms to 33 ms), i.e., removing the frame drop during sensor resolution reconfiguration. Banner also halves the end-to-end resolution reconfiguration latency (226 ms to 105 ms). This enables a more than 49% reduction of system power consumption by allowing continuous vision applications to reconfigure the sensor resolution to 480p compared with downsampling from 1080p to 480p, as measured in a cloud-based offloading workload running on a Jetson TX2 board. As a result, Banner unlocks unprecedented capabilities for mobile vision applications to dynamically reconfigure sensor resolutions to balance the energy efficiency and task accuracy tradeoff.

Mobile vision systems would benefit from the ability to situationally sacrifice image resolution to save system energy when imaging detail is unnecessary. Unfortunately, any change in sensor resolution leads to a substantial pause in frame delivery -- as much as 280 ms. Frame delivery is bottlenecked by a sequence of reconfiguration procedures and memory management in current operating systems before it resumes at the new resolution. This latency from reconfiguration impedes the adoption of otherwise beneficial resolution-energy tradeoff mechanisms. We propose Banner as a media framework that provides a rapid sensor resolution reconfiguration service as a modification to common media frameworks, e.g., V4L2. Banner completely eliminates the frame-to-frame reconfiguration latency (226 ms to 33 ms), i.e., removing the frame drop during sensor resolution reconfiguration. Banner also halves the end-to-end resolution reconfiguration latency (226 ms to 105 ms). This enables a more than 49% reduction of system power consumption by allowing continuous vision applications to reconfigure the sensor resolution to 480p compared with downsampling from 1080p to 480p, as measured in a cloud-based offloading workload running on a Jetson TX2 board. As a result, Banner unlocks unprecedented capabilities for mobile vision applications to dynamically reconfigure sensor resolutions to balance the energy efficiency and task accuracy tradeoff.

Mixed reality mobile platforms attempt to co-locate virtual scenes with physical environments, towards creating immersive user experiences. However, to create visual harmony between virtual and physical spaces, the virtual scene must be accurately illuminated with realistic lighting that matches the physical environment. To this end, we design GLEAM, a framework that provides robust illumination estimation in real-time by integrating physical light-probe estimation with current mobile AR systems. GLEAM visually observes reflective objects to compose a realistic estimation of physical lighting. Optionally, GLEAM can network multiple devices to sense illumination from different viewpoints and compose a richer estimation to enhance realism and fidelity. Using GLEAM, AR developers gain the freedom to use a wide range of materials, which is currently limited by the unrealistic appearance of materials that need accurate illumination, such as liquids, glass, and smooth metals. Our controlled environment user studies across 30 participants reveal the effectiveness of GLEAM in providing robust and adaptive illumination estimation over commercial status quo solutions, such as pre-baked directional lighting and ARKit 2.0 illumination estimation. Our benchmarks reveal the need for situation driven tradeoffs to optimize for quality factors in situations requiring freshness over quality and vice-versa. Optimizing for different quality factors in different situations, GLEAM can update scene illumination as fast as 30ms by sacrificing richness and fidelity in highly dynamic scenes, or prioritize quality by allowing an update interval as high as 400ms in scenes that require high-fidelity estimation.

Mixed reality mobile platforms attempt to co-locate virtual scenes with physical environments, towards creating immersive user experiences. However, to create visual harmony between virtual and physical spaces, the virtual scene must be accurately illuminated with realistic lighting that matches the physical environment. To this end, we design GLEAM, a framework that provides robust illumination estimation in real-time by integrating physical light-probe estimation with current mobile AR systems. We present a demo implementation of GLEAM by means of an AR application that estimates environmental illumination and renders the scene with real-time illumination updates. We demonstrate the efficacy of GLEAM's estimation against a current commercial status quo solution, Apple's ARKit, with the same application.

In this workshop, participants will try their hand at a variety of tangible, embodied, and embedded sensing and feedback technologies including vibrotactile instruments, expressive mechatronics, gesturally modulated fields of light, sound, mist; realtime steerable immersive atmospheres. Working through hands-on experience by theme, participants will be introduced to compositional and experimental methodologies. In the second half of the workshop, participants will compose together some simple "ecosystems" using the Synthesis Center's hardware-software media choreography architecture (sc), in the iStage experimental theater-scale blackbox space.

Mixed-reality haptic devices introduce a gateway to otherwise intangible virtual content, creating a life-like immersive experience. Congruent haptic sensation requires faithful integration of visual stimuli and perceived tactile sensation. Unfortunately, current commercial mixed-reality systems are unable to reproduce the physical sensation of fluid vessels, due to the shifting nature of fluid motion. To this end, we introduce SWISH, a novel type of ungrounded mixed-reality system, capable of affording the users a realistic haptic sensation of fluid behavior. We also present solutions to prominent challenges of rendering haptic fluid behavior, especially in coordinate translation and virtual adaptation to physical limitation. Our virtual-to-physical coupling uses Nvidia Flex's Unreal Engine integration, wirelessly controlling a motorized mechanical actuation system housed in a plastic "vessel''. In this paper we discuss the current state of SWISH and present results from our preliminary user study, followed by a description of our future planned phases.

Current VR/AR systems are unable to reproduce the physical sensation of fluid vessels, due to the shifting nature of fluid motion. To this end, we introduce SWISH, an ungrounded mixed-reality interface, capable of affording the users a realistic haptic sensation of fluid behaviors in vessels. The chief mechanism behind SWISH is in the use of virtual reality tracking and motor actuation to actively relocate the center of gravity of a handheld vessel, emulating the moving center of gravity of a handheld vessel that contains fluid. In addition to solving challenges related to reliable and efficient motor actuation, our SWISH designs place an emphasis on reproducibility, scalability, and availability to the maker culture. Our virtual-to-physical coupling uses Nvidia Flex's Unity integration for virtual fluid dynamics with a 3D printed augmented vessel containing a motorized mechanical actuation system. To evaluate the effectiveness and perceptual efficacy of SWISH, we conduct a user study with 24 participants, 7 vessel actions, and 2 virtual fluid viscosities in a virtual reality environment. In all cases, the users on average reported that the SWISH bucket generates accurate tactile sensations for the fluid behavior. This opens the potential for multi-modal interactions with programmable fluids in virtual environments for chemistry education, worker training, and immersive entertainment.

Continuous mobile vision is limited by the inability to efficiently capture image frames and process vision features. This is largely due to the energy burden of analog readout circuitry, data traffic, and intensive computation. To promote efficiency, we shift early vision processing into the analog domain. This results in RedEye, an analog convolutional image sensor that performs layers of a convolutional neural network in the analog domain before quantization. We design RedEye to mitigate analog design complexity, using a modular column-parallel design to promote physical design reuse and algorithmic cyclic reuse. RedEye uses programmable mechanisms to admit noise for tunable energy reduction. Compared to conventional systems, RedEye reports an 85% reduction in sensor energy, 73% reduction in cloudlet-based system energy, and a 45% reduction in computation-based system energy.

Emerging wearable devices promise a multitude of computer vision-based applications that serve users without active engagement. However, vision algorithms are known to be resource-hungry; and modern mobile systems do not support concurrent application use of the camera. Toward supporting efficient concurrency of vision applications, we report Starfish, a split-process execution system that supports concurrent vision applications by allowing them to share computation and memory objects in a secure and efficient manner. Starfish splits the vision library from an application into a separate process, called the Core, which centrally serves all vision applications. The Core shares library call results among applications, eliminating redundant computation and memory use. Starfish supports unmodified applications and unmodified libraries without needing their source code, and guarantees correctness to the applications. In doing so, Starfish improves both the performance and energy efficiency of concurrent vision applications. Using a prototype implementation on Google Glass, we experimentally demonstrate that Starfish reduces the time spent processing repeated vision library calls by 71% - 97%. When running two to ten concurrent face recognition applications at 0.3 frames per second, Starfish reduces CPU utilization by more than 42% - 80%. Notably, this keeps CPU utilization below 13%, even as the number of applications increases. This reduces system power consumption by 19% - 58%, as Starfish maintains a power consumption at approximately 1210 mW while running the concurrent application workloads.

While computer vision algorithms and libraries have enabled and accelerated the adoption of vision processing into mobile and wearable applications, vision is a resource-hungry operation, and is thus not efficient enough to run on multiple applications simultaneously. However, we observe that many vision algorithms share identical sets of frames and features to perform their analyses, computed from the same library calls. Leveraging this observation, we design a split-process architecture to retrofit existing vision libraries to allow applications to transparently share the computational, memory, and energy overhead of vision processing.

Screen-to-camera visible-light communication links are fundamentally limited by inter-symbol interference, in which the camera receives multiple overlapping symbols in a single capture exposure. By determining interference constraints, we are able to decode symbols with multi-bit depth across all three color channels. We present Styrofoam, a coding scheme which optimally satisfies the constraints by inserting blank frames into the transmission pattern. The coding scheme improves upon the state-of-the-art in camera-based visible-light communication by: (1) ensuring a decode with at least half-exposure of colored multi-bit symbols, (2) limiting decode latency to two transmission frames, and (3) transmitting 0.4 bytes per grid block at the slowest camera's frame rate. In doing so, we outperform peer unsynchronized VLC transmission schemes by 2.9x. Our implementation on smartphone displays and cameras achieves 69.1 kbps.

We report a first-of-its-kind smartphone software system, MoodScope, which infers the mood of its user based on how the smartphone is used. Compared to smartphone sensors that measure acceleration, light, and other physical properties, MoodScope is a "sensor" that measures the mental state of the user and provides mood as an important input to context-aware computing. We run a formative statistical mood study with smartphone-logged data collected from 32 participants over two months. Through the study, we find that by analyzing communication history and application usage patterns, we can statistically infer a user's daily mood average with an initial accuracy of 66%, which gradu-ally improves to an accuracy of 93% after a two-month personal-ized training period. Motivated by these results, we build a service, MoodScope, which analyzes usage history to act as a sensor of the user's mood. We provide a MoodScope API for developers to use our system to create mood-enabled applications. We further create and deploy a mood-sharing social application.

We present MoodScope, a software system which infers the mood of its user based on how the smartphone is used. Similar to smartphone sensors that measure acceleration, light, and other physical properties, MoodScope is a "sensor" that measures the mental state of the user and provides mood as an important input to context-aware computing. We run a formative statistical study with smartphone-logged data collected from 32 participants over two months. Through the study, we find that by analyzing communication history and application usage patterns, we can statistically infer a user's daily mood average with an accuracy of 93% after a two-month training period. Motivated by these results, we build a service, MoodScope, which analyzes usage history to act as a sensor of the user's mood.

A major hurdle to frequently performing mobile computer vision tasks is the high power consumption of image sensing. In this work, we report the first publicly known experimental and analytical characterization of CMOS image sensors. We find that modern image sensors are not energy-proportional: energy per pixel is in fact inversely proportional to frame rate and resolution of image capture, and thus image sensor systems fail to provide an important principle of energy-aware system design: trading quality for energy efficiency. We reveal two energy-proportional mechanisms, supported by current image sensors but unused by mobile systems: (i) using an optimal clock frequency reduces the power up to 50% or 30% for low-quality single frame (photo) and sequential frame (video) capturing, respectively; (ii) by entering low-power standby mode between frames, an image sensor achieves almost constant energy per pixel for video capture at low frame rates, resulting in an additional 40% power reduction. We also propose architectural modifications to the image sensor that would further improve operational efficiency. Finally, we use computer vision benchmarks to show the performance and efficiency tradeoffs that can be achieved with existing image sensors. For image registration, a key primitive for image mosaicking and depth estimation, we can achieve a 96% success rate at 3 FPS and 0.1 MP resolution. At these quality metrics, an optimal clock frequency reduces image sensor power consumption by 36% and aggressive standby mode reduces power consumption by 95%.

A hurdle to frequently performing mobile computer vision tasks is the high energy cost of image sensing. In particular, modern image sensors are not energy proportional; for low resolution and low frame rate capture, the image sensor consumes almost the same amount of energy as it does at high resolutions and high frame rates. We reveal two system-level energy proportional mechanisms: (i) using an optimal pixel clock frequency; (ii) entering low power standby mode between frames. These techniques can be implemented by the image sensor driver with minimal hardware adjustment. Further improvements can be made by designing sensors with heterogeneous hardware architectures. With energy proportionality, computer vision frameworks can be optimized for power consumption, continuously requesting low resolution frames with low energy while only occasionally using high energy to request high resolution frames. This will in turn enable low power continuous mobile vision applications.

To accomplish frequent, simple tasks with high efficiency, it is necessary to leverage low-power, microcontroller-like processors that are increasingly available on mobile systems. However, existing solutions require developers to directly program the low-power processors and carefully manage inter-processor communication. We present Reflex, a suite of compiler and runtime techniques that significantly lower the barrier for developers to leverage such low-power processors. The heart of Reflex is a software Distributed Shared Memory (DSM) that enables shared memory objects with release consistency among code running on loosely coupled processors. In order to achieve high energy efficiency without sacrificing performance much, the Reflex DSM leverages (i) extreme architectural asymmetry between low-power processors and powerful central processors, (ii) aggressive compile-time optimization, and (iii) a minimalist runtime that supports efficient message passing and event-driven execution. We report a complete realization of Reflex that runs on a TI OMAP4430-based development platform as well as on a custom tri-processor mobile platform. Using smartphone sensing applications reported in recent literature, we show that Reflex supports a programming style very close to contemporary smartphone programming. Compared to message passing, the Reflex DSM greatly reduces efforts in programming heterogeneous smartphones, eliminating up to 38% of the source lines of application code. Compared to running the same applications on existing smartphones, Reflex reduces the average system power consumption by up to 81%.