While Augmented Reality (AR) promises to transform remote collaboration, many aspects remain underexplored, particularly where to place remote avatars in messy, everyday environments. Two mixed-methods within-subjects studies examined avatar placement preferences during cooperative (brainstorming) and competitive (negotiation) tasks between participant pairs, focusing on the influence of physical objects (chairs, box, tree) on user preferences. Results showed a strong preference for frontal or slightly off-centre avatar placements, independent of task type. Participants preferred avatar placements that mirrored real-life behaviour, with chairs inviting placements and the tree deterring them. Notably, the large and visually simple box elicited mixed reactions, being viewed alternately as an obstacle to avoid when placing avatars or as an inviting physical anchor for them, despite causing a clear physicality conflict. We term this the "(Anti-)Affordance Problem", highlighting the complexity of avatar placement within physical contexts, and the necessity for AR collaboration platforms to respond to real-world constraints, offering flexibility in avatar placements to accommodate diverse user preferences.

Developing new machine learning applications often requires the collection of new datasets. However, existing datasets may already contain relevant information to train models for new purposes. We propose SoundCollage: a framework to discover new classes within audio datasets by incorporating (1) an audio pre-processing pipeline to decompose different sounds in audio samples, and (2) an automated model-based annotation mechanism to identify the discovered classes. Furthermore, we introduce the clarity measure to assess the coherence of the discovered classes for better training new downstream applications. Our evaluations show that the accuracy of downstream audio classifiers within discovered class samples and a held-out dataset improves over the baseline by up to 34.7% and 4.5%, respectively. These results highlight the potential of SoundCollage in making datasets reusable by labeling with newly discovered classes. To encourage further research in this area, we open-source our code at github.com/nokia-bell-labs/audio-class-discovery.

Sensing human motions through Inertial Measurement Units (IMUs) embedded in personal devices has enabled significant applications in health and wellness. Labeled IMU data is scarce, however, unlabeled or weakly labeled IMU data can be used to model human motions. For video or text modalities, the "pretrain and adapt" approach utilizes large volumes of unlabeled or weakly labeled data to build a strong feature extractor, followed by adaptation to specific tasks using limited labeled data. However, pretraining methods are poorly understood for IMU data, and pipelines are rarely evaluated on out-of-domain tasks. We propose PRIMUS: a method for PRetraining IMU encoderS that uses a novel pretraining objective that is empirically validated based on downstream performance on both in-domain and out-of-domain datasets. The PRIMUS objective effectively enhances downstream performance by combining self-supervision, multimodal, and nearest-neighbor supervision. With fewer than 500 labeled samples per class, PRIMUS improves test accuracy by up to 15%, compared to state-of-the-art baselines. To benefit the broader community, we have open-sourced our code at github.com/nokia-bell-labs/pretrained-imu-encoders.

The rapid adoption of ear-worn devices (earables) has shown significant potential for continuous health monitoring. Despite their close proximity to the human brain and diverse sensing capabilities, the exploration of earable sensing in relation to cognitive function remains underexplored. Building on theoretical and empirical foundations regarding the interplay between cognitive load, auditory complexity, and changes in hearing characteristics influenced by brain function, this study is the first to leverage earable acoustic sensing to assess cognitive load. We specifically designed auditory tasks to elicit four levels of cognitive load and used otoacoustic emissions (OAEs) to measure cochlear response changes in response to cognitive load. By utilizing both audio content indicating auditory complexity and OAEs reflecting hearing characteristic changes, we designed machine learning pipelines to automate the assessment in a four-class cognitive detection task, achieving an accuracy of 68.88%. This research opens a new pathway for using earable acoustic sensing in monitoring cognitive function and holds great potential for future cognitive augmentation.

In a world dominated by podcasts and audiobooks, maintaining auditory attention is essential, yet lapses in focus are common. Auditory attention is crucial for effective communication and comprehension in a distraction-filled environment, as it enables us to focus on important sounds while avoiding external distractions. This work introduces a novel, imperceptible method for detecting auditory attention using earbuds by monitoring muscle movement within the ear canal. We employ an ultrasound-based sensing technique to track phase changes in reflected signals, detecting muscle vibrations associated with shifts in attention. A preliminary user study reveals significant changes in in-ear signal characteristics when participants switch between auditory and cognitive tasks. We show that our system can classify periods of auditory attention and lack of it with an accuracy of 85.7% and a variance of 0.0033. Our findings pave the way for earables that continuously monitor and enhance auditory attention in real-time.

Photoplethysmography (PPG) is the leading non-invasive technique for monitoring biosignals and cardiovascular health, with widespread adoption in both clinical settings and consumer wearable devices. While machine learning models trained on PPG signals have shown promise, they tend to be task-specific and struggle with generalization. Current research is limited by the use of single-device datasets, insufficient exploration of out-of-domain generalization, and a lack of publicly available models, which hampers reproducibility. To address these limitations, we present PaPaGei, the first open foundation model for PPG signals. The model is pre-trained on over 57,000 hours of data, comprising 20 million unlabeled PPG segments from publicly available datasets. We introduce a novel representation learning approach that leverages domain knowledge of PPG signal morphology across individuals, enabling the capture of richer representations compared to traditional contrastive learning methods. We evaluate PaPaGei against state-of-the-art time-series foundation models and self-supervised learning benchmarks across 20 tasks from 10 diverse datasets, spanning cardiovascular health, sleep disorders, pregnancy monitoring, and wellbeing assessment. Our model demonstrates superior performance, improving classification and regression metrics by 6.3% and 2.9% respectively in at least 14 tasks. Notably, PaPaGei achieves these results while being more data- and parameter-efficient, outperforming models that are 70x larger. Beyond accuracy, we examine model robustness across different skin tones, establishing a benchmark for bias evaluation in future models. PaPaGei can serve as both a feature extractor and an encoder for multimodal models, opening up new opportunities for multimodal health monitoring. Models, data, and code are available at: https://github.com/nokia-bell-labs/papagei-foundation-model

Ubiquitous wearable and mobile devices provide access to a diverse set of data. However, the mobility demand for our devices naturally imposes constraints on their computational and communication capabilities. A solution is to locally learn knowledge from data captured by ubiquitous devices, rather than to store and transmit the data in its original form. In this paper, we develop a federated learning framework, called Centaur, to incorporate on-device data selection at the edge, which allows partition-based training of a deep neural nets through collaboration between constrained and resourceful devices within the multidevice ecosystem of the same user. We benchmark on five neural net architecture and six datasets that include image data and wearable sensor time series. On average, Centaur achieves ~19% higher classification accuracy and ~58% lower federated training latency, compared to the baseline. We also evaluate Centaur when dealing with imbalanced non-iid data, client participation heterogeneity, and different mobility patterns. To encourage further research in this area, we release our code at github.com/nokia-bell-labs/data-centric-federated-learning.

We present a real-time multimodal sensing and feedback system that enables closed-loop interaction using wearable devices. Our implementation leverages OmniBuds—a pair of true wireless stereo (TWS) earbuds equipped with dual 9-axis inertial measurement units (IMUs), optical heart rate sensors, and skin temperature sensors, one set in each ear. These sensors continuously stream motion and physiological data via Bluetooth Low Energy (BLE) to a mobile computing platform, where lightweight inference models classify user actions and assess physiological states. Based on this analysis, the system provides real-time auditory feedback through the same earbuds, enabling responsive, human-in-the-loop interaction. As a demonstration, we apply this system to an interactive control scenario based on the T-Rex Chrome Dino game, where users control the avatar using head and body motion captured by the earbuds. Jumping and ducking are recognized in real time through IMU signals, while heart rate and skin temperature dynamically modulates the game speed. The system delivers auditory guidance via the OmniBuds' speakers to help users adapt their actions during game-play. This framework demonstrates the potential of multimodal wearable sensing and closed-loop feedback for embodied interaction, real-time behavioral adaptation, and health-aware interactive systems.

Wearable technology is advancing health monitoring by enabling real-time, privacy-preserving physiological analysis. However, traditional devices often rely on cloud processing, restricting access to raw sensor data and limiting the progress of health-related research. To overcome these limitations, we introduce OmniBuds, a programmable earable research platform that enables multimodal bio-sensing and on-device learning while providing direct access to raw physiological data, fostering advancements in health perception and wearable intelligence. It integrates PPG, temperature, IMUs, and multiple microphones, leveraging an embedded ML accelerator for efficient real-time processing. This paper presents its design, architecture, and applications, demonstrating its potential to shape the future of health-aware wearables.

The rapid growth of wearable devices has opened exciting opportunities for context-aware multi-device collaboration, where multiple devices can provide enhanced user experience tailored to user needs and conditions. However, it also presents a unique challenge of reliably determining whether a set of wearables is being used by the same individual. In real-world scenarios, device sharing, exchanging, or unintended use can cause privacy risks and degraded functionality. Existing solutions primarily rely on accelerometer data to match movement patterns across devices, but they perform poorly during stationary or varied non-repetitive activities. In this paper, we introduce BioQ, a method that unobtrusively detects wearable co-location by generating and matching bio-cues. These bio-cues are generated from on-body wearable sensor data and embedded into a common latent space. Furthermore, when devices share the same sensor types, BioQ can effectively integrate multiple sensor sources to improve cue generation and matching. Experimental results show that BioQ outperforms baselines in bio-cue generation and matching and is resource-effective in model training, inference, and energy use. Our code is available at https://github.com/Nokia-Bell-Labs/contextual-biological-cues.

The advent of tiny artificial intelligence (AI) accelerators enables AI to run at the extreme edge, offering reduced latency, lower power cost, and improved privacy. When integrated into wearable devices, these accelerators open exciting opportunities, allowing various AI apps to run directly on the body. We present Synergy that provides AI apps with besteffort performance via system-driven holistic collaboration over AI accelerator-equipped wearables. To achieve this, Synergy provides device-agnostic programming interfaces to AI apps, giving the system visibility and controllability over the app's resource use. Then, Synergy maximizes the inference throughput of concurrent AI models by creating various execution plans for each app considering AI accelerator availability and intelligently selecting the best set of execution plans. Synergy further improves throughput by leveraging parallelization opportunities over multiple computation units. Our evaluations with 7 baselines and 8 models demonstrate that, on average, Synergy achieves a 23.0× improvement in throughput, while reducing latency by 73.9% and power consumption by 15.8%, compared to the baselines

Overlapping cameras offer exciting opportunities to view a scene from different angles, allowing for more advanced, comprehensive and robust analysis. However, existing video analytics systems for multi-camera streams are mostly limited to (i) per-camera processing and aggregation and (ii) workload-agnostic centralized processing architectures. In this paper, we present Argus, a distributed video analytics system withcross-camera collaborationon smart cameras. We identify multi-camera, multi-target tracking as the primary task of multi-camera video analytics and develop a novel technique that avoids redundant, processing-heavy identification tasks by leveraging object-wise spatio-temporal association in the overlapping fields of view across multiple cameras. We further develop a set of techniques to perform these operations across distributed cameras without cloud support at low latency by (i) dynamically ordering the camera and object inspection sequence and (ii) flexibly distributing the workload across smart cameras, taking into account network transmission and heterogeneous computational capacities. Evaluation of three real-world overlapping camera datasets with two Nvidia Jetson devices shows that Argus reduces the number of object identifications and end-to-end latency by up to 7.13× and 2.19× (4.86× and 1.60× compared to the state-of-the-art), while achieving comparable tracking quality.

Distortion Product OtoAcoustic Emissions (DPOAEs) represents faint cochlear responses to dual-frequency stimuli, commonly employed in hearing screening. This paper introduces an innovative approach to trigger DPOAEs using single-speaker earbuds. Due to their compact size, the speakers used in the earbuds exhibit nonlinear behavior, leading to Inter-Modulation Distortions (IMDs) that interfere with DPOAE signals. Conventional medical devices employ dual speakers to mitigate this distortion, such a solution is impractical for space-constrained earbuds. To address this challenge, we propose a method that triggers DPOAEs while circumventing IMDs by designing a stimulus signal that alternates between the two frequencies necessary for triggering DPOAEs. The performance of our system was evaluated through a preliminary user study involving 8 participants, and it demonstrated a median correlation of 0.65 when compared to a medical-grade reference device.

Self-supervised learning (SSL) has become the de facto training paradigm of large models, where pre-training is followed by supervised fine-tuning using domain-specific data and labels. Despite demonstrating comparable performance with supervised methods, comprehensive efforts to assess SSL's impact on machine learning fairness (i.e., performing equally on different demographic breakdowns) are lacking. Hypothesizing that SSL models would learn more generic, hence less biased representations, this study explores the impact of pre-training and fine-tuning strategies on fairness. We introduce a fairness assessment framework for SSL, comprising five stages: defining dataset requirements, pre-training, fine-tuning with gradual unfreezing, assessing representation similarity conditioned on demographics, and establishing domain-specific evaluation processes. We evaluate our method's generalizability on three real-world human-centric datasets (i.e., MIMIC, MESA, and GLOBEM) by systematically comparing hundreds of SSL and fine-tuned models on various dimensions spanning from the intermediate representations to appropriate evaluation metrics. Our findings demonstrate that SSL can significantly improve model fairness, while maintaining performance on par with supervised methods-exhibiting up to a 30% increase in fairness with minimal loss in performance through self-supervision. We posit that such differences can be attributed to representation dissimilarities found between the best- and the worst-performing demographics across models-up to x13 greater for protected attributes with larger performance discrepancies between segments. Code: https://github.com/Nokia-Bell-Labs/SSLfairness

Tiny machine learning (TinyML) aims to run ML models on small devices and is increasingly favored for its enhanced privacy, reduced latency, and low cost. Recently, the advent of tiny AI accelerators has revolutionized the TinyML field by significantly enhancing hardware processing power. These accelerators, equipped with multiple parallel processors and dedicated per-processor memory instances, offer substantial performance improvements over traditional microcontroller units (MCUs). However, their limited data memory often necessitates downsampling input images, resulting in accuracy degradation. To address this challenge, we propose Data channel EXtension (DEX), a novel approach for efficient CNN execution on tiny AI accelerators. DEX incorporates additional spatial information from original images into input images through patch-wise even sampling and channel-wise stacking, effectively extending data across input channels. By leveraging underutilized processors and data memory for channel extension, DEX facilitates parallel execution without increasing inference latency. Our evaluation with four models and four datasets on tiny AI accelerators demonstrates that this simple idea improves accuracy on average by 3.5%p while keeping the inference latency the same on the AI accelerator. The source code is available at https://github.com/Nokia-Bell-Labs/data-channel-extension.

Self-supervised learning (SSL) has shown remarkable performance in computer vision tasks when trained offline. However, in a Continual Learning (CL) scenario where new data is introduced progressively, models still suffer from catastrophic forgetting. Retraining a model from scratch to adapt to newly generated data is time-consuming and inefficient. Previous approaches suggested re-purposing self-supervised objectives with knowledge distillation to mitigate forgetting across tasks, assuming that labels from all tasks are available during fine-tuning. In this paper, we generalize self-supervised continual learning in a practical setting where available labels can be leveraged in any step of the SSL process. With an increasing number of continual tasks, this offers more flexibility in the pre-training and fine-tuning phases. With Kaizen1, we introduce a training architecture that is able to mitigate catastrophic forgetting for both the feature extractor and classifier with a carefully designed loss function. By using a set of comprehensive evaluation metrics reflecting different aspects of continual learning, we demonstrated that Kaizen significantly outperforms previous SSL models in competitive vision benchmarks, with up to 16.5% accuracy improvement on split CIFAR-100. Kaizen is able to balance the trade-off between knowledge retention and learning from new data with an end-to-end model, paving the way for practical deployment of continual learning systems.

Limited availability of labeled data for machine learning on multimodal time-series extensively hampers progress in the field. Self-supervised learning (SSL) is a promising approach to learn data representations without relying on labels. However, existing SSL methods require expensive computations of negative pairs and are typically designed for single modalities, which limits their versatility. We introduce CroSSL (Cross-modal SSL), which puts forward two novel concepts: masking intermediate embeddings produced by modality-specific encoders, and their aggregation into a global embedding through a cross-modal aggregator CroSSL allows for handling missing modalities and end-to-end cross-modal earning without requiring prior data preprocessing for handling missing inputs or negative-pair sampling for contrastive learning. We evaluate our method on a wide range of data, including motion sensors such as accelerometers or gyroscopes and biosignals (heart rate, electroencephalograms, electromyograms, electrooculograms, and electrodermal). Overall, CroSSL outperforms previous SSL and supervised benchmarks using minimal labeled data, and also sheds light on how latent masking can improve cross-modal learning.

OBJECTIVES: Large language models (LLMs) have demonstrated remarkable generalization and across diverse tasks, leading individuals to increasingly use them as personal assistants due to their emerging reasoning capabilities. Nevertheless, a notable obstacle emerges when including numerical/temporal data into these prompts, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. This article discusses the challenges of representing and tokenizing temporal data. It argues that naively passing timeseries to LLMs can be ineffective due to the modality gap between numbers and text. MATERIALS AND METHODS: We conduct a case study by tokenizing a sample mobile sensing dataset using the OpenAI tokenizer. We also review recent works that feed timeseries data into LLMs for human-centric tasks, outlining common experimental setups like zero-shot prompting and few-shot learning. RESULTS: The case study shows that popular LLMs split timestamps and sensor values into multiple nonmeaningful tokens, indicating they struggle with temporal data. We find that preliminary works rely heavily on prompt engineering and timeseries aggregation to "ground" LLMs, hinting that the "modality gap" hampers progress. The literature was critically analyzed through the lens of models optimizing for expressiveness versus parameter efficiency. On one end of the spectrum, training large domain-specific models from scratch is expressive but not parameter-efficient. On the other end, zero-shot prompting of LLMs is parameter-efficient but lacks expressiveness for temporal data. DISCUSSION: We argue tokenizers are not optimized for numerical data, while the scarcity of timeseries examples in training corpora exacerbates difficulties. We advocate balancing model expressiveness and computational efficiency when integrating temporal data. Prompt tuning, model grafting, and improved tokenizers are highlighted as promising directions. CONCLUSION: We underscore that despite promising capabilities, LLMs cannot meaningfully process temporal data unless the input representation is addressed. We argue that this paradigm shift in how we leverage pretrained models will particularly affect the area of biomedical signals, given the lack of modality-specific foundation models.

This paper presents a novel cancellation method of Intermodulation Distortions (IMDs) for earbud speakers used to measure Distortion Product Otoacoustic Emissions (DPOAE). Speakers’ non-linear behaviour is a significant problem for earbuds with small loudspeakers due to limitations in cone movement. Linear and non-linear speaker modelling enables us to inject exact distortion inverse to cancel what is introduced by speakers’ non-linearities. Our proposed method is compared against state-of-the-art related works in terms of harmonic reduction ratio. Simulation results and evaluation on real hardware show a 77% to 95% reduction in the harmonic distortions of a focused frequency region at the output of the loudspeaker, outperforming existing works by 6% to 14%.

We present SensiX++, a multi-tenant runtime for adaptive model execution with integrated MLOps on edge devices, e.g., a camera, a microphone, or IoT sensors. SensiX++ operates on two fundamental principles: highly modular componentisation to externalise data operations with clear abstractions and document-centric manifestation for system-wide orchestration. First, a data coordinator manages the lifecycle of sensors and serves models with correct data through automated transformations. Next, a resource-aware model server executes multiple models in isolation through model abstraction, pipeline automation, and feature sharing. An adaptive scheduler then orchestrates the best-effort executions of multiple models across heterogeneous accelerators, balancing latency and throughput. Finally, microservices with REST APIs serve synthesised model predictions, system statistics, and continuous deployment. Collectively, these components enable SensiX++ to serve multiple models efficiently with fine-grained control on edge devices while minimising data operation redundancy, managing data and device heterogeneity, and reducing resource contention. We benchmark SensiX++ with 10 different vision and acoustics models across various multi-tenant configurations on different edge accelerators (Jetson AGX and Coral TPU) designed for sensory devices. We report on the overall throughput and quantified benefits of various automation components of SensiX++ and demonstrate its efficacy in significantly reducing operational complexity and lowering the effort to deploy, upgrade, reconfigure, and serve embedded models on edge devices.

WiFi channel state information (CSI) has emerged as a plausible modality for sensing different human vital signs, i.e., respiration and body motion, as a function of modulated wireless signals that travel between WiFi devices. Although a remarkable proposition, most of the existing research in this space struggles to withstand robust performance beyond experimental conditions. To this end, we take a careful look at the dynamics of WiFi signals under human respiration and body motions in the wild. We first characterize the WiFi signal components—multipath and signal subspace—that are modulated by human respiration and body motions. We extrapolate on a set of transformations, including first-order differentiation, max-min normalization and component projections, that faithfully explains and quantifies the dynamics of respiration and body motions on WiFi signals. Grounded in this characterization, we propose two methods: 1) a respiration tracking technique that models the peak dynamics observed in the time-varying signal subspace and 2) a body-motion tracking technique built with a multi-dimensional clustering of evolving signal subspace. Finally, we reflect on the manifestation of these techniques in a practical sleep monitoring application. Our systematic evaluation with over 550 hours of data from 5 users covering both line-of-sight (LOS) and non-line-of-sight (NLOS) settings shows that the proposed techniques can achieve comparable performance to purpose-built pulse-Doppler radar.

Multiple sensory devices on and around us are on the rise and require us to redesign a system to make an inference of ML models accurate, robust, and efficient at the deployment time. While this multiplicity opens up an exciting opportunity to leverage sensor redundancy, it is still extremely challenging to benefit from such multiplicity and boost the runtime performance of deployed ML models without model retraining and engineering. From our experience, we uncovered two prime caveats, device and data variabilities, that affect the runtime performance of ML models. We develop an ML system that addresses these variabilities without modifying deployed models by building on prior algorithmic work. It decouples model execution from sensor data and employs two essential operations between them: a) device-to-device data translation for principled mapping of training and inference data and b) quality-aware dynamic selection of the execution pipeline as a function of runtime accuracy. We evaluate the system on wearable devices with motion and audio-based models. The results show that ML models achieve a 7-13% increase in runtime accuracy solely by running on our system, and the increase goes up to 30% in dynamic environments, at the expense of 3 mW on the host device.

Billions of distributed, heterogeneous, and resource constrained IoT devices deploy on-device machine learning (ML) for private, fast, and offline inference on personal data. On-device ML is highly context dependent and sensitive to user, usage, hardware, and environment attributes. This sensitivity and the propensity toward bias in ML makes it important to study bias in on-device settings. Our study is one of the first investigations of bias in this emerging domain and lays important foundations for building fairer on-device ML. We apply a software engineering lens, investigating the propagation of bias through design choices in on-device ML workflows. We first identifyreliability biasas a source of unfairness and propose a measure to quantify it. We then conduct empirical experiments for a keyword spotting task to show how complex and interacting technical design choices amplify and propagatereliability bias. Our results validate that design choices made during model training, like the sample rate and input feature type, and choices made to optimize models, like light-weight architectures, the pruning learning rate, and pruning sparsity, can result in disparate predictive performance across male and female groups. Based on our findings, we suggest low effort strategies for engineers to mitigate bias in on-device ML.

Continuous blood pressure monitoring is the key to mitigate significant risks for stroke, heart failure and coronary artery disease. Current gold-standard blood pressure devices cause discomfort and interfere with users’ activities. This paper explores an earable system, which continuously monitors users’ blood pressure from the ear. We propose a measurement technique based on the vascular transit time which utilises the time difference between the S1 heart sound and the PPG upstroke in one pulse cycle. We develop a multi-modal sensing hardware and processing pipeline and we evaluate it with 10 participants showing average errors in line with the range recommended by the Association for the Advancement of Medical Instrumentation: 4.07 mmHg for systolic and 5.61 mmHg for diastolic blood pressure.

Sleep deficiency and disorders are one of the most unsolved public health challenges of modern times. Music therapy is a promising approach, offering a cheap and non-invasive solution to improve sleep quality. However, the choice of therapeutic sleep music is highly limited for users because such music needs to be specially chosen and made by sleep therapists. It could potentially lead to the inefficiency of music therapy if users get bored after listening to the same set of music repeatedly. In this paper, we take the first step towards generating personalized sleep therapy music. Firstly, through an in-depth feature analysis, we investigate the importance of various musical and acoustic features of therapy music. Grounded on our findings, we design a style transfer framework called SleepGAN which induces therapeutic features into music from different genres. We show that, compared to baselines, the music generated by SleepGAN has a higher similarity to the sleep music designed by experts.

Federated learning is generally used in tasks where labels are readily available (e.g., next word prediction). Relaxing this constraint requires design of unsupervised learning techniques that can support desirable properties for federated training: robustness to statistical/systems heterogeneity, scalability with number of participants, and communication efficiency. Prior work on this topic has focused on directly extending centralized self-supervised learning techniques, which are not designed to have the properties listed above. To address this situation, we propose Orchestra, a novel unsupervised federated learning technique that exploits the federation’s hierarchy to orchestrate a distributed clustering task and enforce a globally consistent partitioning of clients’ data into discriminable clusters. We show the algorithmic pipeline in Orchestra guarantees good generalization performance under a linear probe, allowing it to outperform alternative techniques in a broad range of conditions, including variation in heterogeneity, number of clients, participation ratio, and local epochs.

Tsetlin Machine (TM) is a new machine learning algorithm that encodes propositional logic into learning automata---a set of logical expressions composed of boolean input features---to recognise patterns. The simplicity, efficiency, and accuracy of this logic-based algorithm encourage rethinking the application of traditional arithmetic-based neural networks (NNs) in intelligent sensors design. Indeed, TM is a promising candidate for embedding intelligence into tiny batteryless sensors with the potential to address two critical challenges: (1) computing under resource constraints and (2) demand for dynamic adaptation to the unpredictable nature of harvested energy. However, its structural model complexity manifests in two conflicting issues: large memory footprint and long latency. This paper addresses these shortcomings by proposing adaptive compression techniques exploiting the inherent redundancies observed in trained models. Through dynamically scaling the computational complexity based on available energy, our techniques significantly reduce the memory footprint and speed up the runtime execution. We evaluate our techniques against standard TMs and binarized neural networks (BNNs) for vision and acoustic workloads deployed on a TI MSP430 MCU operating under intermittent power supply conditions. We show that our techniques can achieve up to 99% compression of TM models and offer 13.5× latency and energy reductions when compared with the most efficient neural network configuration without compromising accuracy.

The development of edge devices with dedicated hardware accelerators has pushed the deployment and inference of Deep Neural Network (DNN) models closer to users and real-world sensory systems than ever before (e.g., wearables, IoT). Recently, a further subset of these devices has emerged: ultra-low power DNN accelerators. These microcontrollers possess a dedicated hardware accelerator and are able to operate with only μJ's of energy in milliseconds of time. With their small form-factor, such devices could be used for battery-powered machine learning (ML) applications. In this work, we take a close look at one such device: the MAX78000 by Maxim Integrated. We characterize the device's performance by running five DNN models of various sizes and architectures, and analyze its operational latency, power consumption, and memory footprint. To better understand the performance characteristics, we take a step further and investigate how different layer types (operation type, kernel size, number of input and output channels) and the selection of accelerator processors affect the execution time.

Breakthroughs in unsupervised domain adaptation (uDA) can help in adapting models from a label-rich source domain to unlabeled target domains. Despite these advancements, there is a lack of research on how uDA algorithms, particularly those based on adversarial learning, can work in distributed settings. In real-world applications, target domains are often distributed across thousands of devices, and existing adversarial uDA algorithms – which are centralized in nature – cannot be applied in these settings. To solve this important problem, we introduce FRuDA: an end-to-end framework for distributed adversarial uDA. Through a careful analysis of the uDA literature, we identify the design goals for a distributed uDA system and propose two novel algorithms to increase adaptation accuracy and training efficiency of adversarial uDA in distributed settings. Our evaluation of FRuDA with five image and speech datasets show that it can boost target domain accuracy by up to 50% and improve the training efficiency of adversarial uDA by at least$11\times$.

We are witnessing a trend of users owning multiple data-generating wearable and IoT devices that continuously capture sensor data pertaining to a user's activities and context. Federated Learning is a potential technique to derive meaningful insights from this sensor data in a privacy-preserving way without revealing the raw sensor data to a central server. In this paper, we introduce a new problem setting in this multi-device context called Federated Learning in Multi-Device Local Networks (FL-MDLN). We identify core challenges for FL-MDLN in relation to its federation architecture, and statistical and systems heterogeneity across multiple users and multiple devices. Then, we introduce a new user-as-client (UAC) federation architecture, and propose various device selection strategies to counter statistical and systems heterogeneity in FL-MDLN. Early empirical findings show that our proposed techniques improve model test accuracy as well as battery power efficiency in FL. Based on these findings, we elucidate open research questions and future work in FL-MDLN.

When deploying machine learning (ML) models on embedded and IoT devices, performance encompasses more than an accuracy metric: inference latency, energy consumption, and model fairness are necessary to ensure reliable performance under heterogeneous and resource-constrained operating conditions. To this end, prior research has studied model-centric approaches, such as tuning the hyperparameters of the model during training and later applying model compression techniques to tailor the model to the resource needs of an embedded device. In this paper, we take a data-centric view of embedded ML and study the role that pre-processing parameters in the data pipeline can play in balancing the various performance metrics of an embedded ML system. Through an in-depth case study with audio-based keyword spotting (KWS) models, we show that pre-processing parameter tuning is a remarkable tool that model developers can adopt to trade-off between a model's accuracy, fairness, and system efficiency, as well as to make an embedded ML model resilient to unseen deployment conditions.

Video cameras are becoming ubiquitous in our daily lives. With the recent advancement of Artificial Intelligence (AI), live video analytics are enabling various useful services, including traffic monitoring and campus surveillance. However, current video analytics systems are highly limited in leveraging the enormous opportunities of the deployed cameras due to (i) centralized processing architecture (i.e., cameras are treated as dumb streaming-only sensors), (ii) hard-coded analytics capabilities from tightly coupled hardware and software, (iii) isolated and fragmented camera deployment from different service providers, and (iv) independent processing of camera streams without any collaboration. In this paper, we envision a full-fledged system for software-defined video analytics with cross-camera collaboration that overcomes the aforementioned limitations. We illustrate its detailed system architecture, carefully analyze the key system requirements with representative app scenarios, and derive potential research issues along with a summary of the status quo of existing works.

This paper introduces a new dataset, Libri-Adapt, to support unsupervised domain adaptation research on speech recognition models. Built on top of the LibriSpeech corpus, Libri-Adapt contains 7200 hours of English speech recorded on mobile and embedded-scale microphones, and spans 72 different domains that are representative of the challenging practical scenarios encountered by ASR models. More specifically, Libri-Adapt facilitates the study of domain shifts in ASR models caused by a) different acoustic environments, b) variations in speaker accents, c) previously unexplored factors such as heterogeneity in the hardware and platform software of the microphones, and d) a combination of the aforementioned three shifts. We also provide a number of baseline results quantifying the impact of these domain shifts on the Mozilla DeepSpeech2 ASR model.

Conversational agents are rich in content today. However, they are entirely oblivious to users’ situational context, limiting their ability to adapt their response and interaction style. To this end, we explore the design space for a context augmented conversational agent, including analysis of input segment dynamics and computational alternatives. Building on these, we propose a solution that redesigns the input segment intelligently for ambient context recognition, achieved in a two-step inference pipeline. We first separate the non-speech segment from acoustic signals and then use a neural network to infer diverse ambient contexts. To build the network, we curated a public audio dataset through crowdsourcing. Our experimental results demonstrate that the proposed network can distinguish between 9 ambient contexts with an average F1 score of 0.80 with a computational latency of 3 milliseconds. We also build a compressed neural network for on-device processing, optimised for both accuracy and latency. Finally, we present a concrete manifestation of our solution in designing a context-aware conversational agent and demonstrate use cases.

For long, we have studied tiny energy harvesters to liberate sensors from batteries. With remarkable progress in embedded deep learning, we are now re-imagining these sensors as intelligent compute nodes. Naturally, we are approaching a crossroad where sensor intelligence is meeting energy autonomy enabling maintenance-free swarm intelligence and unleashing a plethora of applications ranging from precision agriculture to ubiquitous asset tracking to infrastructure monitoring. One of the critical challenges, however, is to adapt intelligence fidelity in response to available energy to maximise the overall system availability. To this end, we present the design and implementation of ePerceptive: a novel framework for best-effort embedded intelligence, i.e., inference fidelity varies in proportion to the instantaneous energy supplied. ePerceptive operates on two core principles. First, it enables training a single deep neural network (DNN) to operate on multiple input resolutions without compromising accuracy or incurring memory overhead. Second, it modifies a DNN architecture by injecting multiple exits to guarantee valid, albeit lower-fidelity inferences in the event of energy interruption. The combination of these techniques offers a smooth adaptation between inference latency and recognition accuracy while matching the computational load to the available power budget. We report the manifestation of ePerceptive in designing batteryless cameras and microphones built with TI MSP430 MCU and off-the-shelf RF and solar energy harvesters. Our evaluation of these batteryless sensors with multiple vision and acoustic workloads suggest that the dynamic adaptation of ePerceptive can increase the inference throughput by up to 80% compared to a static baseline while ensuring a maximum accuracy drop of less than 6%.

Conversational agents are increasingly becoming digital partners of our everyday computing experiences offering a variety of purposeful information and utility services. Although rich on competency, these agents are entirely oblivious to their users' situational and emotional context today and incapable of adjusting their interaction style and tone contextually. To this end, we present a mixed-method study that informs the design of a situation- and emotion-aware conversational agent for kinetic earables. We surveyed 280 users, and qualitatively interviewed 12 users to understand their expectation from a conversational agent in adapting the interaction style. Grounded on our findings, we develop a first-of-its-kind emotion regulator for a conversational agent on kinetic earable that dynamically adjusts its conversation style, tone, volume in response to users emotional, environmental, social and activity context gathered through speech prosody, motion signals and ambient sound. We describe these context models, the end-to-end system including a purpose-built kinetic earable and their real-world assessment. The experimental results demonstrate that our regulation mechanism invariably elicits better and affective user experience in comparison to baseline conditions in different real-world settings.

We explore a new variability observed in motion signals acquired from modern wearables. Wearing variability refers to the variations of the device orientation and placement across wearing events. We collect the accelerometer data on a smartwatch and an earbud and analyse how motion signals change due to the wearing variability. Our analysis shows that the wearing variability can bring an unexpected change to motion signals, not only from different users but also from different wearing sessions of the same user. We also provide empirical ranges of changes in device orientations.

Deep learning has enabled personal and IoT devices to rethink microphones as a multi-purpose sensor for understanding conversation and the surrounding environment. This resulted in a proliferation of Voice Controllable Systems (VCS) around us. The increasing popularity of such systems is also prone to attracting miscreants, who often want to take advantage of the VCS without the knowledge of the user. Consequently, understanding the robustness of VCS, especially under adversarial attacks, has become an important research topic. Although there exists some previous work on audio adversarial attacks, their scopes are limited to embedding the attacks onto pre-recorded music clips, which when played through speakers cause VCS to misbehave. As an attack-audio needs to be played, the occurrence of this type of attacks can be suspected by a human listener. In this paper, we focus on audio-based Denial-of-Service (DoS) attack, which is unexplored in the literature. Contrary to previous work, we show that adversarial audio attacks in real-time and overthe-air are possible, while a user interacts with VCS. We show that the attacks are effective regardless of the user's command and interaction timings. In this paper, we present a first-of-itskind imperceptible and always-on universal audio perturbation technique that enables such DoS attack to be successful. We thoroughly evaluate the performance of the attacking scheme across (i) two learning tasks, (ii) two model architectures and (iii) three datasets. We demonstrate that the attack can introduce as high as 78% error rate in audio recognition tasks.

Unsupervised domain adaptation is emerging as a powerful technique to improve the generalizability of deep learning models to new image domains without using any labeled data in the target domain. In the literature, solutions which perform cross-domain feature-matching (e.g., ADDA), pixel-matching (CycleGAN), and combination of the two (e.g., CyCADA) have been proposed for unsupervised domain adaptation. Many of these approaches make a strong assumption that the source and target label spaces are the same, however in the real-world, this assumption does not hold true. In this paper, we propose a novel solution, FlexAdapt, which extends the state-of-the-art unsupervised domain adaptation approach of CyCADA to scenarios where the label spaces in source and target domains are only partially overlapped. Our solution beats a number of state-of-the-art baseline approaches by as much as 29% in some scenarios, and represent a way forward for applying domain adaptation techniques in the real world.

Mobile and embedded devices are increasingly using microphones and audio-based computational models to infer user context. A major challenge in building systems that combine audio models with commodity microphones is to guarantee their accuracy and robustness in the real-world. Besides many environmental dynamics, a primary factor that impacts the robustness of audio models is microphone variability. In this work, we propose Mic2Mic - a machine-learned system component - which resides in the inference pipeline of audio models and at real-time reduces the variability in audio data caused by microphone-specific factors. Two key considerations for the design of Mic2Mic were: a) to decouple the problem of microphone variability from the audio task, and b) put minimal burden on end-users to provide training data. With these in mind, we apply the principles of cycle-consistent generative adversarial networks (CycleGANs) to learn Mic2Mic using unlabeled and unpaired data collected from different microphones. Our experiments show that Mic2Mic can recover between 66% to 89% of the accuracy lost due to microphone variability for two common audio tasks.

Conversational agents are increasingly becoming digital partners of our everyday computing experiences offering a variety of purposeful information and utility services. Although rich on competency, these agents are entirely oblivious to their users' situational and emotional context today and incapable of adjusting their interaction style and tone contextually. To this end, we present a first-of-its-kind situation-aware conversational agent on kinetic earable that dynamically adjusts its conversation style, tone, volume in response to users emotional, environmental, social and activity context gathered through speech prosody, ambient sound and motion signatures.

The increasing availability of multiple sensory devices on or near a human body has opened brand new opportunities to leverage redundant sensory signals for powerful sensing applications. For instance, personal-scale sensory inferences with motion and audio signals can be done individually on a smartphone, a smartwatch, and even an earbud - each offering unique sensor quality, model accuracy, and runtime behaviour. At execution time, however, it is incredibly challenging to assess these characteristics to select the best device for accurate and resource-efficient inferences. To this end, we look at a quality-aware collaborative sensing system that actively interplays across multiple devices and respective sensing models. It dynamically selects the best device as a function of model accuracy at any given context. We propose two complementary techniques for the runtime quality assessment. Borrowing principles from active learning, our first technique runs on three heuristic-based quality assessment functions that employ confidence, margin sampling, and entropy of models' output. Our second technique is built with a siamese neural network and acts on the premise that runtime sensing quality can be learned from historical data. Our evaluation across multiple motion and audio datasets shows that our techniques provide 12% increase in overall accuracy through dynamic device selection at the average expense of 13 mW power on each device as compared to traditional single-device approaches.

A small variation in mobile hardware and software can potentially cause a significant heterogeneity or variation in the sensor data each device collects. For example, the microphone and accelerometer sensors on different devices can respond very differently to the same audio or motion phenomena. Other factors, like the instantaneous computational load on a smartphone, can cause key behavior like sensor sampling rates to fluctuate, further polluting the data. When sensing devices are deployed in unconstrained and real-world conditions, examples of sharply lower classification accuracy are observed due to what is collectively known as the sensing system heterogeneity. In this work, we take an unconventional approach and argue against solving individual forms of heterogeneity, e.g., improving OS behavior, or the quality/uniformity of components. Instead, we propose and build classifiers that themselves are more tolerant of these variations by leveraging deep learning and a data-augmented training process. Neither augmentation nor deep learning has previously been attempted to cope with sensor heterogeneity. We systematically investigate how these two machine learning methodologies can be adapted to solve such problems, and identify when and where they are able to be successful. We find that our proposed approach is able to reduce classifier errors on an average by 9% and 17% for a range of inertial-and audio-based mobile classification tasks.

No abstract available.

No abstract available.

No abstract available.

We present eSense - an open and multi-sensory in-ear wearable platform for personal-scale behaviour analytics. eSense is a true wireless stereo (TWS) earbud and supports dual-mode Bluetooth and Bluetooth Low Energy. It is also augmented with a 6-axis in-ertial measurement unit and a microphone. We demonstrate the eSense platform, the data exploration tool with the open APIs for the real-time visualisation of multi-modal sensory data, and its manifestation in a 360° workplace well-being application.

360° videos are a new kind of medium that gives the viewers a sense of real immersion as they glimpse the action from all angles and directions. Naturally, professional and amateur film-makers are actively adopting this new medium for transformative storytelling. Despite this phenomenal progress in 360° video creation, current understanding on users' viewing experience of these videos is limited. In this paper, we present the first comparative study on the user experience with 360° videos on mobile devices using different interaction techniques. We observed 18 participants' interaction with six 360°videos with different viewport characteristics (static or moving) on a smartphone, a tablet and a head mounted display (HMD) respectively and measured how they interact with the content. We then conducted semi-structured interviews with the participants in which they explained their interaction with and viewing experience of 360° videos across three devices. Our findings show that 360° videos with moving viewports elicit higher engagement from the viewers, and offer superior viewing experience. However, these videos are cognitively demanding and require constant user attention. Our participants preferred the condition with dynamic peephole interaction on a smartphone for watching 360° videos due to the simplicity in exploration and familiarity with navigation controls. Many participants reported that the HMD offers the most immersive experience however it comes at the expense of higher cognitive burden, motion sickness and physical discomfort.

This paper reports a study on the characterization of in-home Internet activity behavior based on application usage logs. We collected online activity data from 86 Belgium households for 60 days. We analyzed the activity traces to gain insights on the temporal traffic distribution, interaction regularity, and activity correlations. This analysis is then used to develop a generic method to segment households into designated groups showing similar behavioral profiles. Our technique combines interaction frequencies and regularities across activities for segmentation, and is able to reveal interesting time-slotted profile for each segment. These profiles aim to show the strength of routine behaviors in Internet usage, based on which we present a novel algorithm to predict future Internet activities of a household. Our algorithm shows that 60% of the households online activities can be predicted accurately 70% of times.

Wearable devices with built-in cameras present interesting opportunities for users to capture various aspects of their daily life and are potentially also useful in supporting users with low vision in their everyday tasks. However, state-of-the-art image wearables available in the market are limited to capturing images periodically and do not provide any real-time analysis of the data that might be useful for the wearers. In this paper, we present DeepEye - a match-box sized wearable camera that is capable of running multiple cloud-scale deep learn- ing models locally on the device, thereby enabling rich analysis of the captured images in near real-time without offloading them to the cloud. DeepEye is powered by a commodity wearable processor (Snapdragon 410) which ensures its wearable form factor. The software architecture for DeepEye addresses a key limitation with executing multiple deep learning models on constrained hardware, that is their limited runtime memory. We propose a novel inference software pipeline that targets the local execution of multiple deep vision models (specifically, CNNs) by interleaving the execution of computation-heavy convolutional layers with the loading of memory-heavy fully-connected layers. Beyond this core idea, the execution framework incorporates: a memory caching scheme and a selective use of model compression techniques that further minimizes memory bottlenecks. Through a series of experiments, we show that our execution framework outperforms the baseline approaches significantly in terms of inference latency, memory requirements and energy consumption.

Industrial events and exhibitions play a powerful role in creating social relations amongst individuals and firms, enabling them to expand their social network so to acquire resources. However, often these events impose a spatial structure which impacts encounter opportunities. In this paper, we study the impact that the spatial configuration has on the formation of network relations. We designed, developed and deployed a Wi-Fi analytics solution comprising of wearable Wi-Fi badges and gateways in a large scale industrial exhibition event to study the spatio-temporal trajectories of the 2.5K+ attendees including two special groups: 34 investors and 27 entrepreneurs. Our results suggest that certain zones with designated functionalities play a key role in forming social ties across attendees and the different behavioural properties of investors and entrepreneurs can be explained through a spatial lens. Based on our findings we offer three concrete recommendations for future organisers of networking events.

In this paper, we examine the potential of using mobile context to model user engagement. Taking an experimental approach, we systematically explore the dynamics of user engagement with a smartphone through three different studies. Specifically, to understand the feasibility of detecting user engagement from mobile context, we first assess an EEG artifact with 10 users and observe a strong correlation between automatically detected engagement scores and user's subjective perception of engagement. Grounded on this result, we model a set of application level features derived from smartphone usage of 10 users to detect engagement of a usage session using a Random Forest classifier. Finally, we apply this model to train a variety of contextual factors acquired from smartphone usage logs of 130 users to predict user engagement using an SVM classifier with a F1-Score of 0.82. Our experimental results highlight the potential of mobile contexts in designing engagement-aware applications and provide guidance to future explorations.

Face-to-face interactions have proven to accelerate team and larger organisation success. Many past research has explored the benefits of quantifying face-to-face interactions for informed workplace management, however to date, little attention has been paid to understand how the feedback on interaction behaviour is perceived at a personal scale. In this paper, we offer a reflection on the automated feedback of personal interactions in a workplace through a longitudinal study. We designed and developed a mobile system that captured, modelled, quantified and visualised face-to-face interactions of 47 employees for 4 months in an industrial research lab in Europe. Then we conducted semi-structured interviews with 20 employees to understand their perception and experience with the system. Our findings suggest that the short-term feedback on personal face-to-face interactions was not perceived as an effective external cue to promote self-reflection and that employees desire long-term feedback annotated with actionable attributes. Our findings provide a set of implications for the designers of future workplace technology and also opens up avenues for future HCI research on promoting self-reflection among employees.

The popularity of wearables continues to rise. However, their functionalities and applications are constrained by the types of sensors that are currently available. Accelerometers and gyroscopes struggle to capture complex user activities. Microphones and image sensors are more powerful but capture privacy sensitive information. Physiological sensors are obtrusive to users since they often require skin contact and must be placed at certain body positions to function. In contrast, radio- based sensing uses wireless radio signals to capture movements of different parts of body caused by human activities and therefore provides a contactless and privacy-preserving approach to detect and monitor human activities. In this paper, we contribute to the search for a new sensing modality for the next generation of wearable devices by exploring the feasibility of radio-based human activity sensing and recognition in the context of wearable setting. We envision radio-based sensing has the potential to fundamentally transform wearables as we currently know them. As the first step to achieve our vision, we have designed and developed HeadScan, a first- of-its-kind wearable for radio-based sensing of a number of human activities that involve head and mouth movements. HeadScan only requires a pair of small antennas placed on the shoulder and collar and one wearable unit worn on the arm or the belt of the user. HeadScan uses the fine-grained CSI measurements extracted from the radio signals and incorporates a radio signal processing pipeline that converts the raw CSI measurements into the targeted human activities. To examine the feasibility and performance of HeadScan, we have collected about 50.5 hours data from seven users. Our wide-range experiments including comparisons to a conventional skin-contact audio-based sensing approach to tracking the same set of head and mouth-related activities highlight the enormous potential of our radio-based sensing approach and provide guidance to future explorations.

Breakthroughs from the field of deep learning are radically changing how sensor data are interpreted to extract the high-level information needed by mobile apps. It is critical that the gains in inference accuracy that deep models afford become embedded in future generations of mobile apps. In this work, we present the design and implementation of DeepX, a software accelerator for deep learning execution. DeepX signif- icantly lowers the device resources (viz. memory, computation, energy) required by deep learning that currently act as a severe bottleneck to mobile adoption. The foundation of DeepX is a pair of resource control algorithms, designed for the inference stage of deep learning, that: (1) decompose monolithic deep model network architectures into unit- blocks of various types, that are then more efficiently executed by heterogeneous local device processors (e.g., GPUs, CPUs); and (2), perform principled resource scaling that adjusts the architecture of deep models to shape the overhead each unit-blocks introduces. Experiments show, DeepX can allow even large-scale deep learning models to execute efficently on modern mobile processors and significantly outperform existing solutions, such as cloud-based offloading.

Deep learning has revolutionized the way sensor measurements are interpreted and application of deep learning has seen a great leap in inference accuracies in a number of fields. However, the significant requirement for memory and computational power has hindered the wide scale adoption of these novel computational techniques on resource constrained wearable and mobile platforms. In this demonstration we present DeepX, a software accelerator for efficiently running deep neural networks and convolutional neural networks on resource constrained embedded platforms, e.g., Nvidia Tegra K1 and Qualcomm Snapdragon 400.

Wearable devices are increasingly becoming mainstream consumer products carried by millions of consumers. However, the potential impact of these devices is currently constrained by fundamental limitations of their built-in sensors. In this paper, we introduce radio as a new powerful sensing modality for wearable devices and propose to transform radio into a mobile sensor of human activities and vital signs. We present BodyScan, a wearable system that enables radio to act as a single modality capable of providing whole-body continuous sensing of the user. BodyScan overcomes key limitations of existing wearable devices by providing a contactless and privacy-preserving approach to capturing a rich variety of human activities and vital sign information. Our prototype design of BodyScan is comprised of two components: one worn on the hip and the other worn on the wrist, and is inspired by the increasingly prevalent scenario where a user carries a smartphone while also wearing a wristband/smartwatch. This prototype can support daily usage with one single charge per day. Experimental results show that in controlled settings, BodyScan can recognize a diverse set of human activities while also estimating the user's breathing rate with high accuracy. Even in very challenging real-world settings, BodyScan can still infer activities with an average accuracy above 60% and monitor breathing rate information a reasonable amount of time during each day.

We offer a reflection on the technology usage for workplace quantification through an in the wild study. Using a prototype Quantified Workplace system equipped with passive and participatory sensing modalities, we collected and visualized different workplace metrics (noise, color, air quality, self reported mood, and self reported activity) in two European offices of a research organization for a period of 4 months. Next we surveyed 70 employees to understand their engagement experience with the system. We then conducted semi-structured interviews with 20 employees in which they explained which workplace metrics are useful and why, how they engage with the system and what privacy concerns they have. Our findings suggest that sense of inclusion acts as the initial incentive for engagement which gradually translates into a habitual routine. We found that incorporation of an anonymous participatory sensing aspect into the system could lead to sustained user engagement. Compared to past studies we observed a shift in the privacy concerns, due to the trust and transparency of our prototype system. We conclude by providing a set of design principles for building future Quantified Workplace systems.

We present the design, implementation and evaluation of a novel human encounter detection framework for measuring and analysing human behaviour in social settings. We propose the use of WiFi probes, management frames of WiFi, that periodically radiate from mobile devices (as proxies for humans), and existing WiFi access points to automatically capture radio signals and detect human copresence. Based on the spatio-temporal properties of this copresence and their interplay we defined a model, borrowing theories from sociology, to detect human encounters -- short-lived, spontaneous human interactions. We evaluated our framework using controlled and in-the-wild experiments yielding a detection performance of 96% and 86% respectively. As such, our framework opens up interesting opportunities for designing proxemic and group applications, as well as conducting large-scale studies in the areas of computational social sciences.

We investigate how technology usage in homes has changed with the increasing prevalence of mobile devices including Tablets and Smart Phones. We logged Internet usage from 86 Belgium households to determine their six most common Internet Activities. Next, we surveyed households about what devices they own, how they share those devices, and which device they use for different Internet activities. We then conducted semi-structured interviews with 18 of 55 households that responded to the survey in which participants explained their device usage patterns and where they use technology in their home. Our findings suggest that the nature of online activity and social context influence device preference. Many participants reported that their Desktop PC is now a special purpose device, which they use only for specific activities such as working from home or online gaming. Compared to past studies, we observed technology use in many more locations in the home, most notably kitchens and bathrooms.

The study of human activity patterns traditionally relies on the continuous tracking of user location. We approach the problem of activity pattern discovery from a new perspective which is rapidly gaining attention. Instead of actively sampling increasing volumes of sensor data, we explore the participatory sensing potential of multiple mobile social networks, on which users often disclose information about their location and the venues they visit. In this paper, we present automated techniques for filtering, aggregating, and processing combined social networking traces with the goal of extracting descriptions of regularly-occurring user activities, which we refer to as “user routines”. We report our findings based on two localized data sets about a single pool of users: the former contains public geotagged Twitter messages, the latter Foursquare check-ins that provide us with meaningful venue information about the locations we observe. We analyze and combine the two datasets to highlight their properties and show how the emergent features can enhance our understanding of users' daily schedule. Finally, we evaluate and discuss the potential of routine descriptions for predicting future user activity and location.

The Internet of Things connects digital information sources with physical objects - which transforms an artifact from being a passive object into a 'thing' that may link to data, store data and even offer data to users. Digital Object Memories (DOMe) comprise hardware and software components, which together provide an open and universal platform for capturing, associating, and interacting with the digital information of connected objects - including storage, documentation and provision of information concerning actions an object is or might be involved in. The goal of this continuation of an established workshop series (predecessor events include DIPSO 2007-09 in conjunction with Ubicomp 2007-09, DOMe in conjunction with Intelligent Environment 2009, DOMe-IoT 2010 in conjunction with Ubicomp 2010, and NOMe-IoT in conjunction with Ubicomp 2011) is to twofold: 1.) initiate a conversation concerning the potential for objects to develop agency; and 2.) explore how data that is associated with an object may leverage real-world actions. Here, DOMe 2012 provides a hybrid interdisciplinary workshop format that will combine traditional presentations and discussion with practice-based experimentation.

Digital Object Memories (DOMes) comprise hardware and software components, which together provide an open and universal platform for capturing and interacting with the digital information of connected objects - including storage, documentation and provision of information concerning actions an object is or might be involved in. We envisage that connected objects equipped with DOMes will be enabled to make suggestions and propositions to human users - which implies that an object may have a level of agency. The latter concept is a striking possibility that may change the way that we perceive, interact, and relate to objects. The goal of this established workshop series is to twofold: 1.) initiate a conversation concerning the potential for objects to develop agency; and 2.) explore how data that is associated with an object may leverage real-world actions. DOMe-IoT 2012 provides a hybrid interdisciplinary workshop format that will combine traditional presentations and discussion with practice-based experimentation.

Informing caregivers by providing them with contextual medical information can significantly improve the quality of patient care activities. However, information flow in hospitals is still tied to traditional manual or digitised lengthy patient record files that are often not accessible while caregivers are attending to patients. Leveraging the proliferation of pervasive awareness technologies (sensors, actuators and mobile displays), recent studies have explored this information presentation aspect borrowing theories from context-aware computing, i.e., presenting subtle information contextually to support the activity at hand. However, the understanding of the information space (i.e., what information should be presented) is still fairly abstruse, which inhibits the deployment of such real-time activity support systems. To this end, this paper first presents situated glyphs, a graphical entity to encode situation specific information, and then presents our findings from an in-situ qualitative study addressing the information space tailored to such glyphs. Applying technology probes using situated glyphs and different glyph display form factors, the study aimed at uncovering the information space pertained to both primary and secondary medical care. Our analysis has resulted in a large set of information types as well as given us deeper insight on the principles for designing future situated glyphs. We report our findings in this paper that we expect would provide a solid foundation for designing future assistive systems to support patient care activities.

Several do-it-yourself projects and toolkits aim to empower people in bringing intelligence into their personal environments. This is particularly challenging because there is still a large gap between people who can imagine how technology could simplify their life and engineers who can actually make things behave smart. To unite both camps, we present a platform that enables ordinary people to become the inventor of their own smart objects, assisted by a community of field experts. Our software, called D-TALE, fosters the creativity of people by giving them a forum where they can express and prototype their innovative ideas. Moreover, D-TALE gives rise to a crowd-sourcing platform where non-technical users can request developers to provide an implementation for their project. Our approach is outlined in a preliminary case study.

No abstract available.

One shortcoming of self-describing smart objects augmented with digital resources is the limitation of output modalities due to their long established physical appearances. To overcome this drawback intangible representations e.g., sound, video projection etc. are usually coupled with the tangible representations of smart objects that enable access and interaction with their value added features. In this paper, we explore two mobile interaction techniques that associate such intangible representation to smart objects using a pico projector augmented camera phone. The first technique utilizes a Magic Lens metaphor applying mobile augmented reality (contextual information is overlaid while looking at a smart object through camera) to uncover and interact with smart objects. The second technique, Personal Projection follows similar mechanisms in discovery and interaction, except information is projected onto the nearest surface. We report the implementation of these two techniques and a comparative qualitative study with three prototype smart object applications. The findings give us deeper insights on the positive and negative aspects of these two techniques and open up a range of stimulating research issues that we discuss in the paper.

We present an architectural framework that provides the foundation for building smart object systems and uses a document centric approach utilizing a profile based artefact framework and a task based application framework. Our artefact framework represents an instrumented physical smart objects as a collection of service profiles and expresses these services in generic documents. Applications for smart objects are expressed as a collection of functional tasks (independent of the implementation) in a corresponding document. A runtime component provides the foundation for mapping these tasks to the corresponding service provider smart objects. There are three primary advantages of our approach- firstly, it allows developers to write applications in a generic way without prior knowledge of the smart objects that could be used by the applications. Secondly, smart object management (locating/accessing/etc.) issues are completely handled by the infrastructure thus application development becomes rapid and simple. Finally, the programming abstraction used in the framework allows extension of functionalities of smart objects and applications very easily. We describe an implemented prototype of our framework and show examples of its use in a real life scenario to illustrate its feasibility.

This paper explores system issues for involving end users in constructing and enhancing a smart home. In support of this involvement we present an infrastructure and a tangible deployment tool. Active participation of users is essential in a domestic environment as it offers simplicity, greater usercentric control, lower deployment costs and better support for personalization. Our proposed infrastructure provides the foundation for end user deployment utilizing a loosely coupled framework to represent an artefact and its augmented functionalities. Pervasive applications are built independently and are expressed as a collection of functional tasks. A runtime component, FedNet maps these tasks to corresponding service provider artefacts. The tangible deployment tool uses FedNet and allows end users to deploy and control artefacts and applications only by manipulating RFID cards. Primary advantages of our approach are two-fold. Firstly, it allows end users to deploy ubicomp systems easily in a Do-it-Yourself fashion. Secondly, it allows developers to write applications and to build augmented artefacts in a generic way regardless of the constraints of the target environment. We describe an implemented prototype and illustrate its feasibility in a real life deployment session by the end users. Our study shows that the end users might be involved in deploying future ubicomp systems if appropriate tools and supporting infrastructure are provided.

We present a deployment tool atop a document centric infrastraucture to support end users in constructing and extending smart object systems. This tool allows an ordinary individuals to deploy, extend and control smart object systems in a Do-It-Yourself (DIY) fashion. We have implemented two version

This paper explores system issues for enabling incremental deployment of pervasive application - the problem of how to deploy and gradually enhance the functionalities of applications in a pervasive environment. We present a system architecture, FedNet that provides the foundation for incremental d

User centric personalization plays an important role for the adoption of proactive applications. However, stipulating system support to facilitate personalization features in proactive applications generically is still an open issue. In this paper we have addressed this particular issue and presented Persona, a tool that enables adding personalization features in proactive applications in a generic manner. A key feature of Persona is portability that allows it to be injected in various pervasive middlewares as a plug-in. Consequently, existing proactive applications can easily be extended with Persona for personalization support. We have discussed the design and implementation rationale behind Persona and shown it's direct implications with two different middlewares and several proactive applications.

This paper presents a framework prototype for sentient environments. The framework provides a generic interface to the applications for interacting with sentient artefacts in a unified way regardless of their type and properties. As a result, application development is fairly simple, rapid and independent from the context-aware environments.