We present MobiFuse, a high-precision depth perception system on mobile devices that combines dual RGB and Time-of-Flight (ToF) cameras. To achieve this, we leverage physical principles from various environmental factors to propose the Depth Error Indication (DEI) modality, characterizing the depth error of ToF and stereo-matching. Furthermore, we employ a progressive fusion strategy, merging geometric features from ToF and stereo depth maps with depth error features from the DEI modality to create precise depth maps. Additionally, we create a new ToF-Stereo depth dataset,RealToF, to train and validate our model. Our experiments demonstrate that MobiFuse excels over baselines by significantly reducing depth measurement errors by up to 77.7%. It also showcases strong generalization across diverse datasets and proves effectiveness in two downstream tasks: 3D reconstruction and 3D segmentation. The demo video of MobiFuse in real-life scenarios is available at the de-identified YouTube link.

While mobile live video analytics apps require end-to-end latency guarantee for responsiveness and immersiveness, achieving consistent low latency is challenging due to complex fluctuations of wireless channel and scene complexity; for example, latency SLO satisfaction rate drops to as low as 26% in commercial 5G MEC platforms. Prior works mostly focus on either app-only (bitrate, DNN adaptation, or GPU allocation) or RAN-only (radio resource allocation) scheduling, with mutual ignorance of the other side resulting in mismatched scheduling decisions and frequent SLO violations. Coordinating the two schedulers is also challenging, as they are run separately by network and cloud operators with disjoint control. We present ARMA, an end-to-end live video analytics system with app-RAN mutual-awareness for high end-to-end latency SLO satisfaction in MEC. We design a mutually-aware decoupled scheduling mechanism on top of RAN Intelligent Controller (RIC) in Open-RAN architecture that fosters cooperative interaction between the two operators' schedulers while preserving operational proprietaries. We prototype an Open RAN-enabled 5G MEC testbed and evaluate ARMA, showing that ARMA achieves 97% SLO satisfaction rate.

Edge deployment of large Vision-Language Models (VLMs) increasingly relies on flash-based weight offloading, where activation sparsification is used to reduce I/O overhead. However, conventional sparsification remains model-centric, selecting neurons solely by activation magnitude and neglecting how access patterns influence flash performance. We present Neuron Chunking, an I/O-efficient sparsification strategy that operates on *chunks*—groups of contiguous neurons in memory—and couples neuron importance with storage access cost. The method models I/O latency through a lightweight abstraction of access contiguity and selects chunks with high utility, defined as neuron importance normalized by estimated latency. By aligning sparsification decisions with the underlying storage behavior, Neuron Chunking improves I/O efficiency by up to 4.65× and 5.76× on Jetson Orin Nano and Jetson AGX Orin, respectively. The code is available at https://github.com/snuhcs/vlm-flash.

Existing 4D Gaussian Splatting (4DGS) methods struggle to accurately reconstruct dynamic scenes, often failing to resolve ambiguous pixel correspondences and inadequate densification in dynamic regions. We address these issues by introducing a novel method composed of two key components: (1) Elliptical Error Clustering and Error Correcting Splat Addition that pinpoints dynamic areas to improve and initialize fitting splats, and (2) Grouped 4D Gaussian Splatting that improves consistency of mapping between splats and represented dynamic objects. Specifically, we classify rendering errors into missing-color and occlusion types, then apply targeted corrections via backprojection or foreground splitting guided by cross-view color consistency. Evaluations on Neural 3D Video and Technicolor datasets demonstrate that our approach significantly improves temporal consistency and achieves state-of-the-art perceptual rendering quality, improving 0.39dB of PSNR on the Technicolor Light Field dataset. Our visualization shows improved alignment between splats and dynamic objects, and the error correction method’s capability to identify errors and properly initialize new splats. Our implementation details and source code are available at https://github.com/tho-kn/cem-4dgs.

We present EgoTAP, a heatmap-to-3D pose lifting method for highly accurate stereo egocentric 3D pose estimation. Severe self-occlusion and out-of-view limbs in egocentric camera views make accurate pose estimation a challenging problem. To address the challenge, prior methods employ joint heatmaps-probabilistic 2D representations of the body pose, but heatmap-to-3D pose conversion still remains an inaccurate process. We propose a novel heatmap-to-3D lifting method composed of the Grid ViT Encoder and the Propagation Network. The Grid ViT Encoder summarizes joint heatmaps into effective feature embedding using self-attention. Then, the Propagation Network estimates the 3D pose by utilizing skeletal information to better estimate the position of obscure joints. Our method significantly outperforms the previous state-of-the-art qualitatively and quantitatively demonstrated by a 23.9% reduction of error in an MPJPE metric. Our source code is available on GitHub.

Spiking neural network (SNN) is studied in multidisciplinary domains to (i) enable order-of-magnitudes energy-efficient AI inference, and (ii) computationally simulate neuroscientific mechanisms. The lack of discrete theory obstructs the practical application of SNN by limiting its performance and nonlinearity support. We present a new optimization-theoretic perspective of the discrete dynamics of spiking neuron. We prove that a discrete dynamical system of simple integrate-and-fire models approximates the subgradient method over unconstrained optimization problems. We practically extend our theory to introduce a novel sign gradient descent (signGD)-based neuronal dynamics that can (i) approximate diverse nonlinearities beyond ReLU, and (ii) advance ANN-to-SNN conversion performance in low time-steps. Experiments on large-scale datasets show that our technique achieve (i) state-of-the-art performance in ANN-to-SNN conversion, and (ii) is first to convert new DNN architectures, e.g., ConvNext, MLP-Mixer, and ResMLP. We publicly share our source code at www.github.com/snuhcs/snn_signgd .

Accurate tracking of hand and object poses is essential for immersive XR applications. However, existing methods often struggle with the stringent requirements of XR environments. We present XRMan, a real-time hand-object pose tracking system designed for in-the-wild scenarios. XRMan uses a drift monitoring module for consistent accuracy, while farthest-point sampling and collider-based optimization streamline iterative optimization and reduce latency. Our system reduces end-to-end latency by 38.9% compared to the baseline, with further improvements expected from the collider-based post-optimization module.

Cloud-based live video analytics with tight latency bound is gaining importance to support emerging applications such as UAVs and augmented reality. However, existing systems often struggle to meet stringent latency constraints under fluctuating network conditions with packet losses and late-arriving packets. We propose a loss-tolerant live video analytics system called Logan, which effectively accepts packet losses while maintaining high accuracy by utilizing the inherent resilience in DNNs. We design i) Codec-aware Inpainting, which accurately recovers the frame error from packet losses ii) Fast-Forward Recovery that prevents the remaining un-recovered error from propagating over future frames indefinitely. Our results show a 3× improvement (33.2%→99.9%) in SLO satisfaction rate compared to the reliable transmission scheme with <1% accuracy drop under a 5% packet loss rate.

Mixed reality devices with near-eye displays unlock new possibilities for innovation and user experiences. Mixed reality applications require a new unified framework that enables seamless analysis of the real world and simulation of realistic virtual content. Designing such a framework faces various challenges, including huge programming efforts of analysis and simulation pipelines, and inconsistencies between real-world and virtual content caused by end-to-end processes across pipelines.

The recent development of DNN and hardware has created new opportunities for mixed-reality applications. These applications demand the ability to analyze the real world and simulate realistic virtual content. However, designing mixed-reality applications faces diverse challenges due to the absence of a unified framework, such as huge programming effort and inconsistencies between the real scene and virtual content induced by end-to-end latency.

Enabling real-time machine learning (ML) model adaptation to previously unseen, but highly specific contexts and environments can vastly extend the capability of mobile and ubiquitous AI systems. Cloud-aided approaches often fall short of meeting the time constraints without assuming pre-acquisition of data. Other existing approaches targeting efficient training on mobile devices focus on the generally complex context-agnostic tasks where achieving the performance of DNNs without proper backpropagation-based training is unlikely. In this work, we introduce a novel approximate forward training scheme to leverage the relationship that the updates to the parameters of a specific linear (and convolutional) layer in each training step are the linear combinations of outputs from the previous layer. Our preliminary results demonstrate the feasibility of this approach on mobile platforms.

We present GradualReality, a novel interface enabling a Cross Reality experience that includes gradual interaction with physical objects in a virtual environment and supports both presence and usability. Daily Cross Reality interaction is challenging as the user’s physical object interaction state is continuously changing over time, causing their attention to frequently shift between the virtual and physical worlds. As such, presence in the virtual environment and seamless usability for interacting with physical objects should be maintained at a high level. To address this issue, we present an Interaction State-Aware Blending approach that (i) balances immersion and interaction capability and (ii) provides a fine-grained, gradual transition between virtual and physical worlds. The key idea includes categorizing the flow of physical object interaction into multiple states and designing novel blending methods that offer optimal presence and sufficient physical awareness at each state. We performed extensive user studies and interviews with a working prototype and demonstrated that GradualReality provides better Cross Reality experiences compared to baselines.

High-resolution depth estimation, with a minimum resolution of$1280\times 960$, is essential for achieving more immersive experiences in on-device 3D vision applications. However, implementing high-resolution solutions on resource-limited mobile devices presents significant challenges, such as the need for additional expensive depth sensors, computation-intensive machine learning models requiring large-scale datasets, or the need for device motion while the target object remains stationary. In this study, we propose HiMoDepth, an efficient training-free high-resolution depth estimation system that utilizes widely-available on-device dual cameras. HiMoDepth consists of two modules: 1) homogenizing the on-device heterogeneous cameras by iteratively cropping the Field-of-Views to make the focal length of the cameras equal and filtering out the out-of-sync frames based on time stamps, and 2) designing a hierarchical mobile GPU-friendly stereo matching method that effectively reduces the latency of stereo matching with high-resolution depth maps by using efficient data layout, reducing the number of memory accesses, and searching the corresponding pixel over a coarse-to-fine hierarchy. We implement HiMoDepth on multiple commodity mobile devices and conduct comprehensive evaluations. Experimental results show that HiMoDepth significantly outperforms the baselines in both accuracy and running speed on mobile devices that support high-resolution depth maps.

Object detection, while being an attractive interaction method for Augmented Reality (AR), is fundamentally error-prone due to the probabilistic nature of the underlying AI models, resulting in sub-optimal user experiences. In this paper, we explore the effect of three game design concepts, Ambiguity, Transparency, and Controllability, to provide better gameplay experiences in AR games that use error-prone object detection-based interaction modalities. First, we developed a base AR pet breeding game, called Bubbleu that uses object detection as a key interaction method. We then implemented three different variants, each according to the three concepts, to investigate the impact of each design concept on the overall user experience. Our user study results show that each design has its own strengths and can improve player experiences in different ways such as decreasing perceived errors (Ambiguity), explaining the system (Transparency), and enabling users to control the rate of uncertainties (Controllability).

Transformer-based models recently reached state-of-the-art single-channel speech separation accuracy; However, their extreme computational load makes it difficult to deploy them in resource-constrained mobile or IoT devices. We thus present Papez, a lightweight and computation-efficient single-channel speech separation model. Papez is based on three key techniques. We first replace the inter-chunk Transformer with small-sized auditory working memory. Second, we adaptively prune the input tokens that do not need further processing. Finally, we reduce the number of parameters through the recurrent transformer. Our extensive evaluation shows that Papez achieves the best resource and accuracy tradeoffs with a large margin. We publicly share our source code at https://github.com/snuhcs/Papez.

Despite the potential of vision-based personal monitoring (e.g., healthcare), private data leakage concerns hinder its wide deployment in personal spaces (e.g., bedrooms). A body of data anonymization designs was proposed throughout image processing and federated learning. They commonly store high-quality images and videos locally, which are anonymized via post-processing before cloud upload. However, the recent IoT camera hacking and local data leakage call for anonymized data at the sensing stage. Also, continuous and pervasive monitoring without blind spots in complicated indoor spaces requires a scalable and economic system. This paper present Mosaic, a vision-based end-to-end action recognition framework that (i) intrinsically achieves data anonymity from the sensing stage and (ii) battery-free operation for blind spot-free continuous monitoring. Mosaic leverages an extremely low resolution (eLR) Near-Infrared (NIR) image sensor with 6 × 10 pixels for video anonymity and RFID-compliant fully-passive tag with four solar cells for real-time eLR video streaming under as low as 50 lux (e.g., deep in the shelf without direct light). This is accompanied by light-weight action recognition neural network for real-time inference (18.4ms on Intel(R) Core i7-8700). Mosaic achieves an average of 98% accuracy on 10 action classes, hitting the balance between data anonymity and high-precision action recognition. By taking advantage of NIR (non-visible) frequency, Mosaic also works in dark without disturbing sleep. Lastly, wildfire detection reaching 20m was demonstrated, showcasing the potential for outdoor monitoring.

Deep learning models have become increasingly prevalent in various domains, necessitating their deployment on resource-constrained devices. Quantization is a promising way to reduce the model complexity in that it keeps model architecture intact and enables the model to operate on specialized hardwares(e.g., NPU, DSP). Input resolution is also essential in making a trade-off between accuracy and computation.

We present FarfetchFusion, a fully mobile live 3D telepresence system. Enabling mobile live telepresence is a challenging problem as it requires i) realistic reconstruction of the user and ii) high responsiveness for immersive experience. We first thoroughly analyze the live 3D telepresence pipeline and identify three critical challenges: i) 3D data streaming latency and compression complexity, ii) computational complexity of volumetric fusion-based 3D reconstruction, and iii) inconsistent reconstruction quality due to sparsity of mobile 3D sensors. To tackle the challenges, we propose a disentangled fusion approach, which separates invariant regions and dynamically changing regions with our low-complexity spatio-temporal alignment technique, topology anchoring. We then design and implement an end-to-end system, which achieves realistic reconstruction quality comparable to existing server-based solutions while meeting the real-time performance requirements (<100 ms end-to-end latency, 30 fps throughput, <16 ms motion-to-photon latency) solely relying on mobile computation capability.

We present Ego3DPose, a highly accurate binocular egocentric 3D pose reconstruction system. The binocular egocentric setup offers practicality and usefulness in various applications, however, it remains largely under-explored. It has been suffering from low pose estimation accuracy due to viewing distortion, severe self-occlusion, and limited field-of-view of the joints in egocentric 2D images. Here, we notice that two important 3D cues, stereo correspondences, and perspective, contained in the egocentric binocular input are neglected. Current methods heavily rely on 2D image features, implicitly learning 3D information, which introduces biases towards commonly observed motions and leads to low overall accuracy. We observe that they not only fail in challenging occlusion cases but also in estimating visible joint positions. To address these challenges, we propose two novel approaches. First, we design a two-path network architecture with a path that estimates pose per limb independently with its binocular heatmaps. Without full-body information provided, it alleviates bias toward trained full-body distribution. Second, we leverage the egocentric view of body limbs, which exhibits strong perspective variance (e.g., a significantly large-size hand when it is close to the camera). We propose a new perspective-aware representation using trigonometry, enabling the network to estimate the 3D orientation of limbs. Finally, we develop an end-to-end pose reconstruction network that synergizes both techniques. Our comprehensive evaluations demonstrate that Ego3DPose outperforms state-of-the-art models by a pose estimation error (i.e., MPJPE) reduction of 23.1% in the UnrealEgo dataset. Our qualitative results highlight the superiority of our approach across a range of scenarios and challenges.

To acquire language, children need rich language input. However, many parents find it difficult to provide children with sufficient language input, which risks delaying their language development. To aid these parents, we design Captivate!, the first system that provides contextual language guidance to parents during play. Our system tracks both visual and spoken language cues to infer targets of joint attention, enabling the real-time suggestion of situation-relevant phrases for the parent. We design our system through a user-centered process with immigrant families—a highly vulnerable yet understudied population—as well as professional speech language therapists. Next, we evaluate Captivate! on parents with children aged 1–3 to observe improvements in responsive language use. We share insights into developing contextual guidance technology for linguistically diverse families1.

We present FlexPatch, a novel mobile system to enable accurate and real-time object detection over high-resolution video streams. A widely-used approach for real-time video analysis is detection-based tracking (DBT), i.e., running the heavy-but-accurate detector every few frames and applying a lightweight tracker for in-between frames. However, the approach is limited for real-time processing of high-resolution videos in that i) a lightweight tracker fails to handle occlusion, object appearance changes, and occurrences of new objects, and ii) the detection results do not effectively offset tracking errors due to the high detection latency. We propose tracking-aware patching technique to address such limitations of the DBT frameworks. It effectively identifies a set of subareas where the tracker likely fails and tightly packs them into a small-sized rectangular area where the detection can be efficiently performed at low latency. This prevents the accumulation of tracking errors and offsets the tracking errors with frequent fresh detection results. Our extensive evaluation shows that FlexPatch not only enables real-time and power-efficient analysis of high-resolution frames on mobile devices but also improves the overall accuracy by 146% compared to baseline DBT frameworks.

The rapid development of deep learning algorithms, as well as innovative hardware advancements, encourages multi-DNN workloads such as augmented reality applications. However, existing mobile inference frameworks like TensorFlow Lite and MNN fail to efficiently utilize heterogeneous processors available on mobile platforms, because they focus on running a single DNN on a specific processor. As mobile processors are too resource-limited to deliver reasonable performance for such workloads by their own, it is challenging to serve multi-DNN workloads with existing frameworks.

We present LIVE, a system enabling a life-immersive Mixed Reality experience. Daily MR usage is challenging in that the user's interaction state with the physical objects continuously change over time, while the immersion and the utility should be supported simultaneously in the process. As many works of blending the virtual and physical world are designed for a single interaction state, they are not enough to support life-immersive MR. We propose the initial design of LIVE that (i) selects the current user's context among the three states of interaction with physical object and (ii) applies the most suitable blending method to balance immersion and the utility.

Dissemination of sensors and advances in techniques (e.g., network) has led to the opportunity for smart home. However, sensor malfunctions and difficult-to-diagnose characteristics hinder robust sensor system operation. Sensor anomaly detection systems for smart home have been proposed, but they target only a few specific types of sensor anomalies of a single sensor. In this work, we propose a sensor anomaly detection method based on Deep Neural Network (DNN), which automatically extracts critical features to detect the anomalies, even for simultaneous sporadic anomalies with complex data patterns. We leverage Hypersphere Classification (HSC) [14], the state-of-the-art DNN-based supervised outlier exposure method. We evaluate our proposed method on a public smart home sensor dataset. Our results show that the performances of the baselines drop up to 54.4% while ours drops up to 1.1%.

In this work, we show the severity of the thermal issue in mobile Extended Reality (XR) applications. We implement three XR applications and run the applications on a mobile device. We compare the device temperature running benchmark and XR applications. We find that long-term multi-DNN inference execution is the main cause of the thermal issue.

Critical periods are phases during which a toddler’s brain develops in spurts. To promote children’s cognitive development, proper guidance is critical in this stage. However, it is not clear whether such a critical period also exists for the training of AI agents. Similar to human toddlers, well-timed guidance and multimodal interactions might significantly enhance the training efficiency of AI agents as well. To validate this hypothesis, we adapt this notion of critical periods to learning in AI agents and investigate the critical period in the virtual environment for AI agents. We formalize the critical period and Toddler-guidance learning in the reinforcement learning (RL) framework. Then, we built up a toddler-like environment with VECA toolkit to mimic human toddlers’ learning characteristics. We study three discrete levels of mutual interaction: weak-mentor guidance (sparse reward), moderate mentor guidance (helper-reward), and mentor demonstration (behavioral cloning). We also introduce the EAVE dataset consisting of 30,000 real-world images to fully reflect the toddler’s viewpoint. We evaluate the impact of critical periods on AI agents from two perspectives: how and when they are guided best in both uni- and multimodal learning. Our experimental results show that both uni- and multimodal agents with moderate mentor guidance and critical period on 1 million and 2 million training steps show a noticeable improvement. We validate these results with transfer learning on the EAVE dataset and find the performance advancement on the same critical period and the guidance.

Social identification: how much individuals psychologically associate themselves with a group has been posited as an essential construct to measure individual and group dynamics. Studies have shown that individuals who identify very differently from their workgroup provide critical cues to the lack of social support or work overloads. However, measuring identification is typically achieved through time-consuming and privacy-invasive surveys. We hypothesize that the extremities in-group norm affects individuals' behaviors, thus more likely to give rise to negative appraisals. As a more convenient and less-invasive technique, we propose a method to predict individuals who are increasingly different in identifying themselves with their working peers using mobility data passively sensed from the WiFi infrastructure. To test our hypothesis, we collected WiFi data of 62 college students over a whole semester. Students provided regular self-reports on their identification towards a workgroup as ground truth. We analyze the contrasts between groups' mobility patterns and build a classification model to determine students who identify very differently from their workgroup. The classifier achieves approximately 80% True Positive Rate (TPR), 73% True negative rate (TNR), and 78% Accuracy (ACC). Such a mechanism can help distinguish students who are more likely to struggle with negative workgroup appraisals and enable interventions to improve their overall team experience.

We present GROOT, a mobile volumetric video streaming system that delivers three-dimensional data to mobile devices for a fully immersive virtual and augmented reality experience. The system design for streaming volumetric videos should be fundamentally different from conventional 2D video streaming systems. First, the amount of data required to deliver the 3D volume is considerably larger than conventional videos with frames of 2D images, even compared to high-resolution 2D or 360° videos. Second, the 3D data representation, which encodes the surface of objects within the volume, is a sparse and unorganized data structure with varying scales, whereas a conventional video is composed of a sequence of images with the fixed-size 2D grid structure. GROOT is a streaming framework with a novel data structure that enables not only real-time transmission and decoding on mobile devices but also continuous on-demand user view adaptation. Specifically, we modify the conventional octree to introduce the independence of leaf nodes with minimal memory overhead, which enables parallel decoding of highly irregular 3D data. We also developed a suite of techniques to compress color information and filter out 3D points outside of a user's view, which efficiently minimizes the data size and decoding cost. Our extensive evaluation shows that GROOT achieves more stable but faster frame rates compared to any previous method to stream and visualize volumetric videos on mobile devices.

We present EagleEye, an AR-based system that identifies missing person (or people) in large, crowded urban spaces. Designing EagleEye involves critical technical challenges for both accuracy and latency. Firstly, despite recent advances in Deep Neural Network (DNN)-based face identification, we observe that state-of-the-art models fail to accurately identify Low-Resolution (LR) faces. Accordingly, we design a novel Identity Clarification Network to recover missing details in the LR faces, which enhances true positives by 78% with only 14% false positives. Furthermore, designing EagleEye involves unique challenges compared to recent continuous mobile vision systems in that it requires running a series of complex DNNs multiple times on a high-resolution image. To tackle the challenge, we develop Content-Adaptive Parallel Execution to optimize complex multi-DNN face identification pipeline execution latency using heterogeneous processors on mobile and cloud. Our results show that EagleEye achieves 9.07X faster latency compared to naive execution, with only 108 KBytes of data offloaded.

We present Heimdall, a mobile GPU coordination platform for emerging Augmented Reality (AR) applications. Future AR apps impose an explored challenging workload: i) concurrent execution of multiple Deep Neural Networks (DNNs) for physical world and user behavior analysis, and ii) seamless rendering in presence of the DNN execution for immersive user experience. Existing mobile deep learning frameworks, however, fail to support such workload: multi-DNN GPU contention slows down inference latency (e.g., from 59.93 to 1181 ms), and rendering-DNN GPU contention degrades frame rate (e.g., from 30 to ≈12 fps). Multi-tasking for desktop GPUs (e.g., parallelization, preemption) cannot be applied to mobile GPUs as well due to limited architectural support and memory bandwidth. To tackle the challenge, we design a Pseudo-Preemption mechanism which i) breaks down the bulky DNN into smaller units, and ii) prioritizes and flexibly schedules concurrent GPU tasks. We prototyped Heimdall over various mobile GPUs (i.e., recent Adreno series) and multiple AR app scenarios that involve combinations of 8 state-of-the-art DNNs. Our extensive evaluation shows that Heimdall enhances the frame rate from ≈12 to ≈30 fps while reducing the worst-case DNN inference latency by up to ≈15 times compared to the baseline multi-threading approach.

This paper tackles the problem of detecting a comprehensive set of sensor faults that can occur in IoT-instrumented smart homes customized to infer Activities of Daily Living (ADL) from the activation of sensor sets. Specifically, sensors can suffer faults that (a) span durations that vary between several seconds to hours, (b) can result in both missing or false-alarm sensor-events. Previous fault detection approaches are geared primarily to identify missing faults (absence of sensor readings) of a permanent (very long-lived) nature, or sporadic false-alarm events. We propose DetectIF, a fault-detection framework that detects faults of varying time duration, and identifies both missing and false-alarm sensor events. DetectIF's key novelties include developing rules capturing spatiotemporal correlations among sensors and augmenting those rules with statistical properties of such sensor-specific behavior. To test DetectIF under a variety of fault behavior, we develop a unified fault framework where the tuning of a couple of parameters allows us to generate and inject faults of desired type and duration into an underlying sensor stream. Experiments with such comprehensive fault data shows that DetectIF achieves 82-95% fault-detection accuracy, improving precision by a huge amount (33-66%) over competitive, state-of-the-art baselines. Moreover, we demonstrate the benefits of applying DetectIF on unmodified, benchmark smart home datasets: it is able to detect additional likely faults that prior fault detection approaches miss, and thus consequently achieve an average of 30% higher ADL recognition accuracy compared to prior state-of-the-art fault detection techniques.

Today's smartphone application (hereinafter `app') markets do not provide information on power consumption of apps, which is essential for users. Continuous sensing apps make this problem more severe because significant power is consumed without the users' awareness. We propose PowerForecaster to break through such an exhaustive cycle. It provides users with personalized estimation of sensing apps' power cost at pre-installation time. It is challenging to provide such estimation in advance because the actual power cost of a sensing app varies depending on user behavior such as physical activities and phone use patterns. To address this, we develop a novel power emulator as a core component of PowerForecaster. It achieves accurate, personalized power estimation by reproducing users' behaviors and emulating the target app's power use. We optimize the system to make the power emulation fast and its trace collection energy efficient. We further address the problem of dealing with large-scale emulation requests from worldwide deployment. We develop a novel selective emulation approach to minimize the server-side resource cost. We performed extensive experiments and the experimental results show that PowerForecaster achieves the power estimation accuracy of 93.4 percent and saves on 60 percent of the emulator instance usage.

With mobile apps rapidly permeating all aspects of daily living with use by all segments of the population, it is crucial to support the evaluation of app usability for specific impaired users to improve app accessibility. In this work, we examine the effects of using our augmented virtuality impairment simulation system--Empath-D--to support experienced designer-developers to redesign a mockup of commonly used mobile application for cataract-impaired users, comparing this with existing tools that aid designing for accessibility. We show that the use of augmented virtuality for assessing usability supports enhanced usability challenge identification, finding more defects and doing so more accurately than with existing methods. Through our user interviews, we also show that augmented virtuality impairment simulation supports realistic interaction and evaluation to provide a concrete understanding over the usability challenges that impaired users face, and complements the existing guidelines-based approaches meant for general accessibility.

Deep-learning (DL) is receiving huge attention as enabling techniques for emerging mobile and IoT applications. It is a common practice to conduct DNN model-based inference using cloud services due to their high computation and memory cost. However, such a cloud-offloaded inference raises serious privacy concerns. Malicious external attackers or untrustworthy internal administrators of clouds may leak highly sensitive and private data such as image, voice and textual data. In this paper, we propose Occlumency, a novel cloud-driven solution designed to protect user privacy without compromising the benefit of using powerful cloud resources. Occlumency leverages secure SGX enclave to preserve the confidentiality and the integrity of user data throughout the entire DL inference process. DL inference in SGX enclave, however, impose a severe performance degradation due to limited physical memory space and inefficient page swapping. We designed a suite of novel techniques to accelerate DL inference inside the enclave with a limited memory size and implemented Occlumency based on Caffe. Our experiment with various DNN models shows that Occlumency improves inference speed by 3.6x compared to the baseline DL inference in SGX and achieves a secure DL inference within 72% of latency overhead compared to inference in the native environment.

It has been a common approach to apply Machine Learning (ML) techniques over sensory data for inferring human behavior, activities, emotions, and surrounding contexts. Especially, IMU (Inertial Measurement Unit) sensors are widely used to obtain dataset to train ML models for human activity recognition. A key challenge in building highly accurate ML models lies in collecting a wide variety of activity data from a large number of users. Such data collection is often a highly time-consuming and costly process.

Much research argues that feeling overwhelmed by stress and for prolonged periods can lead to severe mental illness such as early onset depression and anxiety among many others. Recovering from severe stress to a normal state is much easier, in terms of the length of time and treatment required, compared to when more serious conditions have manifested [1]. Unfortunately, existing stress monitoring applications either require dedicated applications to be installed on the user's mobile device or use various mobile and wearable sensors [2, 4, 6, 7]; thus are not scalable to large number of users. Our goal is to provide a community-wide "safety net" that will automatically and non-intrusively detect individuals exhibiting signs of excessive stress without them installing any dedicated app. YouTube Demo Link https://youtu.be/LKQvIX4W6L0

We present VitaMon, a mobile sensing system that can measure the inter-heartbeat interval (IBI) from the facial video captured by a commodity smartphone's front camera. The continuous IBI measurement is used to compute heart rate variability (HRV), one of the most important markers of the autonomic nervous system (ANS) regulation. The underlying idea of VitaMon is that video recording of human face contains multiple cardiovascular pulse signals with different phase shift. Our measurement on 10 participants shows the significant time delay (36.79 ms) between the pulse signals measured at the jaw region and forehead region. VitaMon leverages deep neural network models to extract both spatial and temporal information of the video to reconstruct a pulse waveform signal that is optimized for estimating IBI. We evaluated VitaMon with a dataset collected from 30 participants under various conditions involving different light intensity levels and motion artifacts. With the 15 fps video input (66.67 ms time resolution), VitaMon can measure IBI with an average error of 14.26 ms and 21.65 ms using personal and general model respectively. HRV features including geometry Poincare plot, time- and frequency-domain features extracted from the IBI measurement all have high correlation with the reference signal.

We present SmrtFridge, a consumer-grade smart fridge prototype that demonstrates two key capabilities: (a) identify the individual food items that users place in or remove from a fridge, and (b) estimate the residual quantity of food items inside a refrigerated container (opaque or transparent). Notably, both of these inferences are performed unobtrusively, without requiring any explicit user action or tagging of food objects. To achieve these capabilities, SmrtFridge uses a novel interaction-driven, multi-modal sensing pipeline, where Infrared (IR) and RGB video sensing, triggered whenever a user interacts naturally with the fridge, is used to extract a foreground visual image of the food item, which is then processed by a state-of-the-art DNN classifier. Concurrently, the residual food quantity is estimated by exploiting slight thermal differences, between the empty and filled portions of the container. Experimental studies, involving 12 users interacting naturally with 19 common food items and a commodity fridge, show that SmrtFridge is able to (a) extract at least 75% of a food item's image in over 97% of interaction episodes, and consequently identify the individual food items with precision/recall values of ~ 85%, and (b) perform robust coarse-grained (3 level) classification of the residual food quantity with an accuracy of ~ 75%.

Stress and depression are a common affliction in all walks of life. When left unmanaged, stress can inhibit productivity or cause depression. Depression can occur independently of stress. There has been a sharp rise in mobile health initiatives to monitor stress and depression. However, these initiatives usually require users to install dedicated apps or multiple sensors, making such solutions hard to scale. Moreover, they emphasise sensing individual factors and overlook social interactions, which plays a significant role in influencing stress and depression while being a part of a social system. We present StressMon, a stress and depression detection system that leverages single-attribute location data, passively sensed from the WiFi infrastructure. Using the location data, it extracts a detailed set of movement, and physical group interaction pattern features without requiring explicit user actions or software installation on client devices. These features are used in two different machine learning models to detect stress and depression. To validate StressMon, we conducted three different longitudinal studies at a university with different groups of students, totalling up to 108 participants. Our evaluation demonstrated StressMon detecting severely stressed students with a 96.01% True Positive Rate (TPR), an 80.76% True Negative Rate (TNR), and a 0.97 area under the ROC curve (AUC) score (a score of 1 indicates a perfect binary classifier) using a 6-day prediction window. In addition, StressMon was able to detect depression at 91.21% TPR, 66.71% TNR, and 0.88 AUC using a 15-day window. We end by discussing how StressMon can expand CSCW research, especially in areas involving collaborative practices for mental health management.

In this paper, we present I4S, a system that identifies item interactions of customers in a retail store through sensor data fusion from smartwatches, smartphones and distributed BLE beacons. To identify these interactions, I4S builds a gesture-triggered pipeline that (a) detects the occurrence of "item picks", and (b) performs fine-grained localization of such pickup gestures. By analyzing data collected from 31 shoppers visiting a midsized stationary store, we show that we can identify person-independent picking gestures with a precision of over 88%, and identify the rack from where the pick occurred with 91%+ precision (for popular racks).

Real-world deployments of WiFi-based indoor localization in large public venues are few and far between as most state-of-the-art solutions require either client or infrastructure-side changes. Hence, even though high location accuracy is possible with these solutions, they are not practical due to cost and/or client adoption reasons. Majority of the public venues use commercial controller-managed WLAN solutions, that neither allow client changes nor infrastructure changes. In fact, for such venues we have observed highly heterogeneous devices with very low adoption rates for client-side apps.

With app-based interaction increasingly permeating all aspects of daily living, it is essential to ensure that apps are designed to be inclusive and are usable by a wider audience such as the elderly, with various impairments (e.g., visual, audio and motor). We propose Empath-D, a system that fosters empathetic design, by allowing app designers, in-situ, to rapidly evaluate the usability of their apps, from the perspective of impaired users. To provide a truly authentic experience, Empath-D carefully orchestrates the interaction between a smartphone and a VR device, allowing the user to experience simulated impairments in a virtual world while interacting naturally with the app, using a real smartphone. By carefully orchestrating the VR-smartphone interaction, Empath-D tackles challenges such as preserving low-latency app interaction, accurate visualization of hand movement and low-overhead perturbation of I/O streams. Experimental results show that user interaction with Empath-D is comparable (both in accuracy and user perception) to real-world app usage, and that it can simulate impairment effects as effectively as a custom hardware simulator.

No abstract available.

We describe the design and implementation of a smartwatch-based, completely unobtrusive, food journaling system, where the smartwatch helps to intelligently capture useful images of food that an individual consumes throughout the day. The overall system, called Annapurna, is based on three key components: (a) a smartwatch-based gesture recognizer to identify eating gestures, (b) a smartwatch-based image capturer that obtains a small set of relevant and useful images with a low energy overhead, and (c) a server-based image filtering engine that removes irrelevant uploaded images, and then catalogs them through a portal. Our primary challenge is to make the system robust to the huge diversity in natural eating habits and food choices. We show how we address this by an appropriate coupling between a smartwatch's camera sensor and inertial sensor-based tracking of eating gestures, thereby helping to capture multiple likely-to-be-useful images with low energy overhead. Through a series of real-world, in-the-wild studies, we demonstrate the end-to-end working of Annapurna, which captures useful images in over 95% of all natural eating episodes.

We present Follow-My-Lead, an alternative indoor navigation technique that uses visual information recorded on an actual navigation path as a navigational guide. Its design revealed a trade-off between the fidelity of information provided to users and their effort to acquire it. Our first experiment revealed that scrolling through a continuous image stream of the navigation path is highly informative, but it becomes tedious with constant use. Discrete image checkpoints require less effort, but can be confusing. A balance may be struck by adding fast video transitions between image checkpoints, but precise control is required to handle difficult situations. Authoring still image checkpoints is also difficult, and this inspired us to invent a new technique using video checkpoints. We conducted a second experiment on authoring and navigation performance and found video checkpoints plus fast video transitions to be better than both image checkpoints plus fast video transitions and traditional written instructions.

We propose BreathPrint, a new behavioural biometric signature based on audio features derived from an individual's commonplace breathing gestures. Specifically, BreathPrint uses the audio signatures associated with the three individual gestures: sniff, normal, and deep breathing, which are sufficiently different across individuals. Using these three breathing gestures, we develop the processing pipeline that identifies users via the microphone sensor on smartphones and wearable devices. In BreathPrint, a user performs breathing gestures while holding the device very close to their nose. Using off-the-shelf hardware, we experimentally evaluate the BreathPrint prototype with 10 users, observed over seven days. We show that users can be authenticated reliably with an accuracy of over 94% for all the three breathing gestures in intra-sessions and deep breathing gesture provides the best overall balance between true positives (successful authentication) and false positives (resiliency to directed impersonation and replay attacks). Moreover, we show that this breathing sound based biometric is also robust to some typical changes in both physiological and environmental context, and that it can be applied on multiple smartphone platforms. Early results suggest that breathing based biometrics show promise as either to be used as a secondary authentication modality in a multimodal biometric authentication system or as a user disambiguation technique for some daily lifestyle scenarios.

No abstract available.

The rapid emergence of head-mounted devices such as the Microsoft Holo-lens enables a wide variety of continuous vision applications. Such applications often adopt deep-learning algorithms such as CNN and RNN to extract rich contextual information from the first-person-view video streams. Despite the high accuracy, use of deep learning algorithms in mobile devices raises critical challenges, i.e., high processing latency and power consumption. In this paper, we propose DeepMon, a mobile deep learning inference system to run a variety of deep learning inferences purely on a mobile device in a fast and energy-efficient manner. For this, we designed a suite of optimization techniques to efficiently offload convolutional layers to mobile GPUs and accelerate the processing; note that the convolutional layers are the common performance bottleneck of many deep learning models. Our experimental results show that DeepMon can classify an image over the VGG-VeryDeep-16 deep learning model in 644ms on Samsung Galaxy S7, taking an important step towards continuous vision without imposing any privacy concerns nor networking cost.

We propose PADA, a new power evaluation tool to measure and optimize power use of mobile sensing applications. Our motivational study with 53 professional developers shows they face huge challenges in meeting power requirements. The key challenges are from the significant time and effort for repetitive power measurements since the power use of sensing applications needs to be evaluated under various real-world usage scenarios and sensing parameters. PADA enables developers to obtain enriched power information under diverse usage scenarios in development environments without deploying and testing applications on real phones in real-life situations. We conducted two user studies with 19 developers to evaluate the usability of PADA. We show that developers benefit from using PADA in the implementation and power tuning of mobile sensing applications.

In this paper, we present LiveLabs, a first-of-its-kind testbed that is deployed across a university campus, convention centre, and resort island and collects real-time attributes such as location, group context etc., from hundreds of opt-in participants. These venues, data, and participants are then made available for running rich human-centric behavioural experiments that could test new mobile sensing infrastructure, applications, analytics, or more social-science type hypotheses that influence and then observe actual user behaviour. We share case studies of how researchers from around the world have and are using LiveLabs, and our experiences and lessons learned from building, maintaining, and expanding Live-Labs over the last three years.

Continuous mobile sensing applications are emerging. Despite their usefulness, their real-world adoption has been slow. Many users are turned away by the drastic battery drain caused by continuous sensing and processing. In this paper, we propose CoMon+, a novel cooperative context monitoring system, which addresses the energy problem through opportunistic cooperation among nearby users. For effective cooperation, we develop a benefit-aware negotiation method to maximize the energy benefit of context sharing. CoMon+ employs heuristics to detect cooperators who are likely to remain in the vicinity for a long period of time, and the negotiation method automatically devises a cooperation plan that provides mutual benefit to cooperators, while considering running applications, available devices, and user policies. Especially, CoMon+ improves the negotiation method proposed in our earlier work, CoMon [30], to exploit multiple processing plans enabled by various personal sensing devices; each plan can be alternatively used for cooperation, which in turn will maximize overall power saving. We implement a CoMon+ prototype and show that it provides significant benefit for mobile sensing applications, e.g., saving 27-71 percent of smartphone power consumption depending on cooperation cases. Also, our deployment study shows that CoMon+ saves an average 19.7 percent of battery under daily use of a prototype application compared to the case without CoMon+ running.

In this paper, we show that characterizing user behaviour from location or smartphone usage traces, without accounting for the interaction of individuals in physical-world groups, can lead to erroneous results. We conducted one of the largest studies in the UbiComp domain thus far, involving indoor location traces of more than 6,000 users, collected over a 4-month period at our university campus, and further studied fine-grained App usage of a subset of 156 Android users. We apply a state-of-the-art group detection algorithm to annotate such location traces with group vs. individual context, and then show that individuals vs. groups exhibit significant differences along three behavioural traits: (1) the mobility pattern, (2) the responsiveness to calls / SMSs and (3) application usage. We show that these significant differences are robust to underlying errors in the group detection technique and that the use of such group context leads to behavioural results that differ from those reported in prior popular work.

Emerging continuous sensing apps introduce new major factors governing phones' overall battery consumption behaviors: (1) added nontrivial persistent battery drain, and more importantly (2) different battery drain rate depending on the user's different mobility condition. In this paper, we address the new battery impacting factors significant enough to outdate users' existing battery model in real life. We explore an initial approach to help users understand the cause and effect between their physical activity and phones' battery life. To this end, we present Sandra, a novel mobility-aware smartphone battery information advisor, and study its potential to help users redevelop their battery model. We perform an extensive explorative study and deployment for 30 days with 24 users. Our findings reveal what they essentially learned, and in which situations they found Sandra very helpful. We share the lessons learned to help in the design of future mobility-aware battery advisors.

We present QueueVadis, a system that addresses the problem of estimating, in real-time, the properties of queues at commonplace urban locations, such as coffee shops, taxi stands and movie theaters. Abjuring the use of any queuing-specific infrastructure sensors, QueueVadis uses participatory mobile sensing to detect both (i) the individual-level queuing episodes for any arbitrarily-shaped queue (by a characteristic locomotive signature of short bursts of "shuffling forward" between periods of "standing") and (ii) the aggregate-level queue properties (such as expected wait or service times) via appropriate statistical aggregation of multi-person data. Moreover, for venues where multiple queues are too close to be separated via location estimates, QueueVadis also uses a novel disambiguation technique to separate users into multiple distinct queues. User studies, performed with 138 cumulative total users observed at 23 different real-world queues across Singapore and Japan, show that QueueVadis is able to (a) identify all individual queuing episodes, (b) predict service and wait times fairly accurately (with median estimation errors in the 10%--20% range), independent of the queue's shape, (c) separate users in multiple proximate queues with close to 80% accuracy and (d) provide reasonable estimates when the participation rate (the fraction of QueueVadis-equipped people in the queue) is modest.

Driven by the rapid market growth of sensors and beacons that offer Bluetooth Low Energy (BLE) based connectivity, this paper empirically investigates the performance characteristics of the BLE interface on multiple Android smartphones, and the consequent impact on a proposed BLE-based service: continuous indoor location. We first use extensive measurement studies with multiple Android devices to establish that the BLE interface on current smartphones is not as "low-energy" as nominally expected, and establish that continuous use of such a BLE interface is not feasible unless we choose a moderately large scan interval and a low duty cycle. We then explore the implications of such constraints, on the parameters of a smart phone's BLE stack, on the accuracy of a BLE-based indoor localization techniques. We show that while RF-based indoor location can be highly accurate (80% of estimates have errors less than or equal to 4 meters) for stationary users only if the density of beacons is high, the combination of (large scan interval, low duty cycle) causes the location error to degrade significantly for moving users. These results provide practical insights into the use cases and limitations for future BLE-based mobile services.

Mobile games, and especially multiplayer games are a very popular daily distraction for many users. We hypothesise that commuters travelling on public buses or trains would enjoy being able to play multiplayer games with their fellow commuters to alleviate the commute burden and boredom. We present quantitative data to show that the typical one-way commute time is fairly long (at least 25 minutes on average) as well as survey results indicating that commuters are willing to play multiplayer games with other random commuters. In this paper, we present GameOn, a system that allows commuters to participate in multiplayer games with each other using p2p networking techniques that reduces the need to use high latency and possibly expensive cellular data connections. We show how GameOn uses a cloud-based matchmaking server to eliminate the overheads of discovery as well as show why GameOn uses Wi-Fi Direct over Bluetooth as the p2p networking medium. We describe the various system components of GameOn and their implementation. Finally, we present numerous results collected by using GameOn, with three real games, on many different public trains and buses with up to four human players in each game play.

In the public studying space, students sometimes hog seats. This behavior prevents others from access their needed seats, and requires man power to patrol and move stuffs from hogged seats to another place. In this paper, we present a capacitive sensor to check seat occupancy statuses in real-time (i.e. un-occupied, people occupying, and seat hogging). The device helps students to find available seats, and helps staffs to monitor seats usage remotely for better controlling over seat hogging problem without the need to patrol. Our experiment results show that the sensor can accurately detect seat occupancy. Moreover, it can clearly distinguish between human occupancy and seat hogging.

Recognizing user emotional states while running entertainment applications such as playing game or watching video is very important to understand and improve user experience. In this work, we designed a practical system using wearable physiological sensors including skin electrical conductivity and photoplethysmography (PPG) to recognize popular non-negative emotions that people experience when they watch videos or play games on their mobile devices. We demonstrates how our system recognizes emotional states in two phases: (1) classifying the levels of arousal (high or low) and valence (positive or neutral) at the accuracy of 94.74% and 78.95% respectively; (2) recognizing three non-negative emotions: excitement, contentment, amusement by mapping arousal and valence levels using Russel circumplex model of affect.

Today's smartphone application (hereinafter 'app') markets miss a key piece of information, power consumption of apps. This causes a severe problem for continuous sensing apps as they consume significant power without users' awareness. Users have no choice but to repeatedly install one app after another and experience their power use. To break such an exhaustive cycle, we propose PowerForecaster, a system that provides users with power use of sensing apps at pre-installation time. Such advanced power estimation is extremely challenging since the power cost of a sensing app largely varies with users' physical activities and phone use patterns. We observe that the time for active sensing and processing of an app can vary up to three times with 27 people's sensor traces collected over three weeks. PowerForecaster adopts a novel power emulator that emulates the power use of a sensing app while reproducing users' physical activities and phone use patterns, achieving accurate, personalized power estimation. Our experiments with three commercial apps and two research prototypes show that PowerForecaster achieves 93.4% accuracy under 20 use cases. Also, we optimize the system to accelerate emulation speed and reduce overheads, and show the effectiveness of such optimization techniques.

Recently, a number of continuous sensing applications have been actively proposed in research communities and commercially released in the market. However, due to their unique power characteristics, user behavior-dependent battery drain, they bring new challenges for users' power management on these applications. In this demonstration, we present a comprehensive approach to support users' power management for continuous sensing applications. First, at pre-installation time, we provide an instant, personalized power estimation of a continuous sensing application. Without exhaustive trial and error, users can decide judiciously to install a certain application or not. Second, at runtime, we provide mobility-aware battery information. With this information, users can better estimate the phone's remaining battery life based on their imminent mobility conditions and take necessary actions in advance such as carrying an additional battery or minimizing the use of applications.

Language delay is a developmental problem of children who do not acquire language as expected for their chronological ages. Without timely intervention, language delay can act as a lifelong risk factor. Speech-language pathologists highlight that effective parent participation in everyday parent-child conversation is important to treat children's language delay. For effective roles, however, parents need to alter their own lifelong-established conversation habits, requiring extensive period of conscious effort and staying alert. In this paper, we present new opportunities for mobile and social computing to reinforce everyday parent-child conversation with therapeutic implications for children with language delays. Specifically, we propose TalkBetter, a mobile in-situ intervention service to help parents in daily parent-child conversation through real-time meta-linguistic analysis of ongoing conversations. Through extensive field studies with speech-language pathologists and parents, we report the multilateral motivations and implications of TalkBetter. We present our development of TalkBetter prototype and report its performance evaluation.

Interpersonal touch is our most primitive social language strongly governing our emotional well-being. Despite the positive implications of touch in many facets of our daily social interactions, we find wide-spread caution and taboo limiting touch-based interactions in workplace relationships that constitute a significant part of our daily social life. In this paper, we explore new opportunities for ubicomp technology to promote a new meme of casual and cheerful interpersonal touch such as high-fives towards facilitating vibrant workplace culture. Specifically, we propose High5, a mobile service with a smartwatch-style system to promote high-fives in everyday workplace interactions. We first present initial user motivation from semi-structured interviews regarding the potentially controversial idea of High5. We then present our smartwatch-style prototype to detect high-fives based on sensing electric skin potential levels. We demonstrate its key technical observation and performance evaluation.

In this paper, we reduce the energy overheads of continuous mobile sensing for context-aware applications that are interested in collective context or events. We propose a cloud-based query management and optimization framework, called CloQue, which can support concurrent queries, executing over thousands of individual smartphones. CloQue exploits correlation across context of different users to reduce energy overheads via two key innovations: i) Dynamically reordering the order of predicate processing to preferentially select predicates with not just lower sensing cost and higher selectivity, but that maximally reduce the uncertainty about other context predicates, and ii) intelligently propagating the query evaluation results to dynamically update the uncertainty of other correlated, but yet-to-be evaluated, context predicates. An evaluation, using real cell phone traces from a real world dataset shows significant energy savings (between 30 to 50% compared with traditional short-circuit systems) with little loss in accuracy (5% at most).

Detecting the group context of an individual (i.e., whether an individual is alone or part of a group) in crowded public spaces, such as shopping malls, is an important goal with many practical applications. However, in crowded indoor spaces, understanding the group-dependent movement behavior is a non-trivial problem as: (1) detecting groups is hard as the density ensures that at any location, a large number of people are moving together, (2) location tracking in many real-world venues is either absent or not very accurate, and (3) indoor mobility models that take into account group attributes (such as group size) are rare. In this paper, we first introduce GruMon, a platform for near real-time group monitoring in dense, public spaces, and then demonstrate how the movement & residency properties of individuals are significantly affected when they are in groups.

Real-time monitoring of groups and their rich contexts will be a key building block for futuristic, group-aware mobile services. In this paper, we propose GruMon, a fast and accurate group monitoring system for dense and complex urban spaces. GruMon meets the performance criteria of precise group detection at low latencies by overcoming two critical challenges of practical urban spaces, namely (a) the high density of crowds, and (b) the imprecise location information available indoors. Using a host of novel features extracted from commodity smartphone sensors, GruMon can detect over 80% of the groups, with 97% precision, using 10 minutes latency windows, even in venues with limited or no location information. Moreover, in venues where location information is available, GruMon improves the detection latency by up to 20% using semantic information and additional sensors to complement traditional spatio-temporal clustering approaches. We evaluated GruMon on data collected from 258 shopping episodes from 154 real participants, in two large shopping complexes in Korea and Singapore. We also tested GruMon on a large-scale dataset from an international airport (containing ≈37K+ unlabelled location traces per day) and a live deployment at our university, and showed both GruMon's potential performance at scale and various scalability challenges for real-world dense environment deployments.

In this paper, we present Orchestrator, an active resource orchestration framework for a PAN-scale sensor-rich mobile computing platform. Incorporating diverse sensing devices connected to a mobile phone, the platform will serve as a common base to accommodate personal context-aware applications. A major challenge for the platform is to simultaneously support concurrent applications requiring continuous and complex context processing, with highly scarce and dynamic resources. To address the challenge, we build Orchestrator, which actively coordinates applications' resource uses over the distributed mobile and sensor devices. As a key approach, it adopts an active resource use orchestration, which prepares multiple alternative plans for application requests and selectively applies them according to resource availability and demands at runtime. Through the selection, it resolves resource contention among applications and helps them efficiently share resources. With such system-level supports, applications become capable of providing long-running services under dynamic circumstances with scarce resources. Also, the platform can host a number of applications stably, exploiting its full resource capacity. We build an Orchestrator prototype on off-the-shelf mobile devices and sensor motes and show its effectiveness in terms of application supportability and resource use efficiency.

In this paper, we propose SocioPhone, a novel initiative to build a mobile platform for face-to-face interaction monitoring. Face-to-face interaction, especially conversation, is a fundamental part of everyday life. Interaction-aware applications aimed at facilitating group conversations have been proposed, but have not proliferated yet. Useful contexts to capture and support face-to-face interactions need to be explored more deeply. More important, recognizing delicate conversational contexts with commodity mobile devices requires solving a number of technical challenges. As a first step to address such challenges, we identify useful meta-linguistic contexts of conversation, such as turn-takings, prosodic features, a dominant participant, and pace. These serve as cornerstones for building a variety of interaction-aware applications. SocioPhone abstracts such useful meta-linguistic contexts as a set of intuitive APIs. Its runtime efficiently monitors registered contexts during in-progress conversations and notifies applications on-the-fly. Importantly, we have noticed that online turn monitoring is the basic building block for extracting diverse meta-linguistic contexts, and have devised a novel volume-topography-based method. We show the usefulness of SocioPhone with several interesting applications: SocioTherapist, SocioDigest, and Tug-of-War. Also, we show that our turn-monitoring technique is highly accurate and energy-efficient under diverse real-life situations.

No abstract available.

Mobile applications that sense continuously, such as location monitoring, are emerging. Despite their usefulness, their adoption in real-world deployment situations has been extremely slow. Many smartphone users are turned away by the drastic battery drain caused by continuous sensing and processing. Also, the extractable contexts from the phone are quite limited due to its position and sensing modalities. In this paper, we propose CoMon, a novel cooperative ambience monitoring platform, which newly addresses the energy problem through opportunistic cooperation among nearby mobile users. To maximize the benefit of cooperation, we develop two key techniques, (1) continuity-aware cooperator detection and (2) benefit-aware negotiation. The former employs heuristics to detect cooperators who will remain in the vicinity for a long period of time, while the latter automatically devises a cooperation plan that provides mutual benefit to cooperators, while considering running applications, available devices, and user policies. Through continuity- and benefit-aware operation, CoMon enables applications to monitor the environment at much lower energy consumption. We implement and deploy a CoMon prototype and show that it provides significant benefit for mobile sensing applications.

No abstract available.

No abstract available.

We envision that diverse social exercising games, or exergames, will emerge, featuring much richer interactivity with immersive game play experiences. Further, the recent advances of mobile devices and wireless networking will make such social engagement more pervasive - people carry portable exergame devices (e.g., jump ropes) and interact with remote users anytime, anywhere. Towards this goal, we explore the potential of using heterogeneous exercise devices as game controllers for a multi-player social exergame; e.g., playing a boat paddling game with two remote exercisers (one with a jump rope, and the other with a treadmill). In this paper, we propose a novel platform called ExerLink that converts exercise intensity to game inputs and intelligently balances intensity/delay variations for fair game play experiences. We report the design considerations and guidelines obtained from the design and development processes of game controllers. We validate the efficacy of game controllers and demonstrate the feasibility of social exergames with heterogeneous exercise devices via extensive human subject studies.

We demonstrate a pervasive social exergame platform called ExerLink that converts exercise intensity to game inputs and intelligently balances intensity/delay variations for fair game play experiences. Also, we show the potential of using multiple exercise devices as game controllers and incorporating multiple heterogeneous controllers into a game. Specifically, we consider a class of exercise equipment used for repetitive, individual, and aerobic (RIA) exercises such as treadmill running, stationary cycling, hula hooping, and jump roping.

In this paper, we propose a novel efficient dataflow execution method for mobile context monitoring applications. As a key approach to minimize the execution overhead, we propose a new dataflow execution model, producer-oriented model. Compared to the conventional consumer-oriented model adopted in stream processing engines, our model significantly reduces execution overhead to process context monitoring dataflow reflecting unique characteristics of context monitoring. To realize the model, we develop DataBank, an execution container that takes charge of the management and delivery of the output data for the associated operator. We demonstrate the effectiveness of DataBank by implementing three useful applications and their dataflow graphs, i.e., MusicMap, FindMyPhone, and CalorieMonitor. Using the applications, we show that DataBank reduces the CPU utilization by more than 50%, compared to the methods based on the consumer-oriented model; DataBank enables more context monitoring applications to run concurrently.

In this demonstration, we will show MobiCon, a context monitoring platform; it runs over smartphones and sensor OSs, and facilitates development and deployment of everyday context-aware applications. For many years, lots of research efforts have been made in building low-cost, yet effective sensor networks for various application domains such as structural health monitoring of bridges, disaster recovery, automated ventilation of buildings. Integration of sensors into smartphones and the advent of wearable devices open a new opportunity for mobile applications to leverage in-situ user contexts such as his/her location, activity, social relationship, health status. In recent studies of mobile and pervasive computing, a number of useful mobile context-aware applications have been proposed, but their actual deployment is slow due to complexity of context processing and heavy resource and battery usage. To address such challenges, we have been building MobiCon for many years, upon which diverse context-aware applications are developed and deployed without concerns about complexity of context processing and resource optimization.

Emerging mobile sensing applications are changing the characteristics of smartphone workloads. Whereas typical mobile applications run alone in the foreground interacting with users, sensing applications concurrently run in the background, providing unobtrusive monitoring services. Such concurrent sensing workloads raise a new challenge incurring severe resource contention among themselves and with other foreground applications. To address the challenge, we develop SymPhoney, a coordinated sensing flow execution engine to support concurrent sensing applications. As its key approach, we develop a novel sensing-flow-aware coordination. We first introduce the new concept of frame externalization i.e., to identify and externalize semantic structures embedded in otherwise flat sensing data streams. Leveraging the identified frame structures, SymPhoney develops frame-based coordination and scheduling mechanisms, which effectively coordinates the resource use of concurrent contending applications and maximize their utilities even under severe resource contention. We implemented several sensing applications on top of the SymPhoney engine and performed extensive experiments, showing effective coordination capability of SymPhoney.

In this paper, we propose an in-situ Crowd-aware Sensing Platform, called "CrowdMon", which envisions the cooperation among mobile users in highly crowded urban areas such as metro and square. CrowdMon establishes a spontaneous connection from co-located users in a semantic proximity and enables them to share contextual information such as location, ambient music, and mood of places. To the best of our knowledge, CrowdMon is the first attempt to support crowd-aware services at a platform level. We show interesting use cases of CrowdMon and an initial system design to realize the crowd-based context sharing.

In this paper, we introduce Activity Travel Pattern (ATP) monitoring in a large-scale city environment. ATP represents where city residents and vehicles stay and how they travel around in a complex megacity. Monitoring ATP will incubate new types of value-added services such as predictive mobile advertisement, demand forecasting for urban stores, and adaptive transportation scheduling. To enable ATP monitoring, we develop ActraMon, a high-performanceATP monitoring framework. As a first step, ActraMon provides a simple but effective computational model of ATP and a declarative query language facilitating effective specification of various ATP monitoring queries. More important, ActraMon employs the shared staging architecture and highly efficient processing techniques, which address the scalability challenges caused by massive location updates, a number of ATP monitoring queries and processing complexity of ATP monitoring. Finally, we demonstrate the extensive performance study of ActraMon using realistic city-wide ATP workloads.

No abstract available.

This paper proposes MISSA, a novel middleware to facilitate the development and provision of stream-based services in emerging pervasive environments. The stream-based services utilize voluminous and continuously updated data streams as their input. The characteristics of data streams bring new requirements on the development and provision of the services. To satisfy the requirements, a unique service model and a runtime system are designed in MISSA. The key concept of our service model is to separate service logic from handling data streams. This significantly mitigates the burden on service developers by allowing them to only concentrate on the service logic. Job of handling data streams is completely delegated to the runtime. In this paper, we first present the importance of stream-based services and their requirements. Also, we describe a best route finding service as an example to motivate the need for MISSA. Then, we envision overall architecture for provisioning of stream-based services and detail our design of service model and runtime. Lastly, we demonstrate the efficiency of service model and runtime through experiments.

In this paper, we present Orchestrator, an active resource orchestration framework for mobile context monitoring. Emerging pervasive environments will introduce a PAN-scale sensor-rich mobile platform consisting of a mobile device and many wearable and space-embedded sensors. In such environments, it is challenging to enable multiple context-aware applications requiring continuous context monitoring to simultaneously run and share highly scarce and dynamic resources. Orchestrator enables multiple applications to effectively share the resources while exploiting the full capacity of overall system resources and providing high-quality service to users. For effective orchestration, we propose an active resource use orchestration approach that actively finds appropriate resource uses for applications and flexibly utilizes them depending on dynamic system conditions. Orchestrator is built upon a prototype platform that consists of off-the-shelf mobile devices and sensor motes. We present the detailed design, implementation, and evaluation of Orchestrator. The evaluation results show that Orchestrator enables applications in a resource-efficient way.

The key feature of many emerging pervasive computing applications is to proactively provide services to mobile individuals. One major challenge in providing users with proactive services lies in continuously monitoring users' context based on numerous sensors in their PAN/BAN environments. The context monitoring in such environments imposes heavy workloads on mobile devices and sensor nodes with limited computing and battery power. We present SeeMon, a scalable and energy-efficient context monitoring framework for sensor-rich, resource-limited mobile environments. Running on a personal mobile device, SeeMon effectively performs context monitoring involving numerous sensors and applications. On top of SeeMon, multiple applications on the mobile device can proactively understand users' contexts and react appropriately. This paper proposes a novel context monitoring approach that provides efficient processing and sensor control mechanisms. We implement and test a prototype system on two mobile devices: a UMPC and a wearable device with a diverse set of sensors. Example applications are also developed based on the implemented system. Experimental results show that SeeMon achieves a high level of scalability and energy efficiency.

In this paper, we present BMQ-Processor, a high-performance border-crossing event (BCE) detection framework for large-scale monitoring applications. We first characterize a new query semantics, namely, border monitoring query (BMQ), which is useful for BCE detection in many monitoring applications. It monitors the values of data streams and reports them only when data streams cross the borders of its range. We then propose BMQ-Processor to efficiently handle a large number of BMQs over a high volume of data streams. BMQ-Processor efficiently processes BMQs in a shared and incremental manner. It develops and operates over a novel stateful query index, achieving a high level of scalability over continuous data updates. Also, it utilizes the locality embedded in data streams and greatly accelerates successive BMQ evaluations. We present data structures and algorithms to support 1D as well as multidimensional BMQs. We show that the semantics of border monitoring can be extended toward more advanced ones and build region transition monitoring as a sample case. Lastly, we demonstrate excellent processing performance and low storage cost of BMQ-Processor through extensive analysis and experiments.

Proactively providing services to mobile individuals is essential for emerging ubiquitous applications. The major challenge in providing users with proactive services lies in continuously monitoring their contexts based on numerous sensors. The context monitoring with rich sensors imposes heavy workloads on mobile devices with limited computing and battery power. We present SeeMon, a scalable and energy-efficient context monitoring framework for sensor-rich, resource-limited mobile environments. Running on a personal mobile device, SeeMon effectively performs context monitoring involving numerous sensors and applications. On top of SeeMon, multiple applications on the device can proactively understand users' contexts and react appropriately. This paper proposes a novel context monitoring approach that provides efficient processing and sensor control mechanisms. We implement and test a prototype system on two mobile devices: a UMPC and a wearable device with a diverse set of sensors. Example applications are also developed based on the implemented system. Experimental results show that SeeMon achieves a high level of scalability and energy efficiency.

Border Monitoring Query (BMQ) has different query semantic from conventional continuous range query. It monitors the values of data streams and reports them only when data streams cross the borders of its range. In this paper, we first emphasize the importance and usefulness of BMQ through attractive service scenarios. Then, we propose BMQ-Index, which is specialized to BMQ evaluation. It efficiently processes a large number of BMQs in a shared and incremental manner. For shared processing, BMQ-Index adopts a query indexing approach, thereby achieving a high level of scalability. For incremental processing, BMQ-Index employs an incremental access method. Thus, successive BMQ evaluations are significantly accelerated. We present an index structure and a search algorithm to support onedimensional as well as multi-dimensional BMQ. Lastly, we demonstrate the performance benefits of BMQ-Index through analysis and experiments.