The miniaturisation of neural processing units (NPUs) and other low-power accelerators has enabled their integration into microcontroller-scale wearable hardware, supporting near-real-time, offline, and privacy-preserving inference. Yet physiological signal analysis has remained infeasible on such hardware; recent Transformer-based models show state-of-the-art performance but are prohibitively large for resource- and power-constrained hardware and incompatible with µNPUs due to their dynamic attention operations. We introduce PhysioLite, a lightweight, NPU-compatible model architecture and training framework for ECG/EMG signal analysis. Using learnable wavelet filter banks, CPU-offloaded positional encoding, and hardware-aware layer design, PhysioLite reaches performance comparable to state-of-the-art Transformer-based foundation models on ECG and EMG benchmarks, while being <10% of the size (∼ 370KB with 8-bit quantization). We also profile its component-wise latency and resource consumption on both the MAX78000 and HX6538 WE2 µNPUs, demonstrating its viability for signal analysis on constrained, battery-powered hardware. We release our model(s) and training framework at: https://github.com/j0shmillar/physiolite.

Developing new machine learning applications often requires the collection of new datasets. However, existing datasets may already contain relevant information to train models for new purposes. We propose SoundCollage: a framework to discover new classes within audio datasets by incorporating (1) an audio pre-processing pipeline to decompose different sounds in audio samples, and (2) an automated model-based annotation mechanism to identify the discovered classes. Furthermore, we introduce the clarity measure to assess the coherence of the discovered classes for better training new downstream applications. Our evaluations show that the accuracy of downstream audio classifiers within discovered class samples and a held-out dataset improves over the baseline by up to 34.7% and 4.5%, respectively. These results highlight the potential of SoundCollage in making datasets reusable by labeling with newly discovered classes. To encourage further research in this area, we open-source our code at github.com/nokia-bell-labs/audio-class-discovery.

Ubiquitous wearable and mobile devices provide access to a diverse set of data. However, the mobility demand for our devices naturally imposes constraints on their computational and communication capabilities. A solution is to locally learn knowledge from data captured by ubiquitous devices, rather than to store and transmit the data in its original form. In this paper, we develop a federated learning framework, called Centaur, to incorporate on-device data selection at the edge, which allows partition-based training of a deep neural nets through collaboration between constrained and resourceful devices within the multidevice ecosystem of the same user. We benchmark on five neural net architecture and six datasets that include image data and wearable sensor time series. On average, Centaur achieves ~19% higher classification accuracy and ~58% lower federated training latency, compared to the baseline. We also evaluate Centaur when dealing with imbalanced non-iid data, client participation heterogeneity, and different mobility patterns. To encourage further research in this area, we release our code at github.com/nokia-bell-labs/data-centric-federated-learning.

Speech Foundation Models encounter significant performance degradation when deployed in real-world scenarios involving acoustic domain shifts, such as background noise and speaker accents. Test-time adaptation (TTA) has recently emerged as a viable strategy to address such domain shifts at inference time without requiring access to source data or labels. However, existing TTA approaches, particularly those relying on backpropagation, are memory-intensive, limiting their applicability in speech tasks and resource-constrained settings. Although backpropagation-free methods offer improved efficiency, existing ones exhibit poor accuracy. This is because they are predominantly developed for vision tasks, which fundamentally differ from speech task formulations, noise characteristics, and model architecture, posing unique transferability challenges. In this paper, we introduce E-BAT, first Efficient BAckpropagation-free TTA framework designed explicitly for speech foundation models. E-BAT achieves a balance between adaptation effectiveness and memory efficiency through three key components: (i) lightweight prompt adaptation for a forward-pass-based feature alignment, (ii) a multi-scale loss to capture both global (utterance-level) and local distribution shifts (token-level) and (iii) a test-time exponential moving average mechanism for stable adaptation across utterances. Experiments conducted on four noisy speech datasets spanning sixteen acoustic conditions demonstrate consistent improvements, with 4.1\%--13.5% accuracy gains over backpropogation-free baselines and 2.0$\times$–6.4$\times$ GPU memory savings compared to backpropogation-based methods. By enabling scalable and robust adaptation under acoustic variability, this work paves the way for developing more efficient adaptation approaches for practical speech processing systems in real-world environments.

Traditional human-in-the-loop-based annotation for time-series data like inertial data often requires access to alternate modalities like video or audio from the environment. These alternate sources provide the necessary information to the human annotator, as the raw numeric data are often too obfuscated even for an expert. However, this traditional approach has many concerns surrounding overall cost, efficiency, storage of additional modalities, time, scalability, and privacy. Interestingly, recent large language models (LLMs) are also trained with vast amounts of publicly available alphanumeric data, which allows them to comprehend and perform well on tasks beyond natural language processing. Naturally, this opens up a potential avenue to explore the opportunities in using these LLMs as virtual annotators where the LLMs will be directly provided the raw sensor data for annotation instead of relying on any alternate modality. Naturally, this could mitigate the problems of the traditional human-in-the-loop approach. Motivated by this observation, we perform a detailed study in this article to assess whether the state-of-the-art (SOTA) LLMs can be used as virtual annotators for labeling time-series physical sensing data. To perform this in a principled manner, we segregate the study into two major phases. In the first phase, we investigate the challenges an LLM like GPT-4 faces in comprehending raw sensor data. Considering the observations from Phase 1, in the next phase, we investigate the possibility of encoding the raw sensor data using SOTA SSL approaches and utilizing the projected time-series data to get annotations from the LLM. Detailed evaluation with four benchmark HAR datasets shows that SSL-based encoding and metric-based guidance allow the LLM to make more reasonable decisions and provide accurate annotations without requiring computationally expensive fine-tuning or sophisticated prompt engineering.

Intelligent city transportation systems are one of the core infrastructures of a smart city. The true ingenuity of such an infrastructure lies in providing the commuters with real-time information about citywide transport like public buses, allowing them to pre-plan their travel. However, providing prior information for transportation systems like public buses in real-time is inherently challenging because of the diverse nature of different stay-locations where a public bus stops. Although straightforward factors like stay duration extracted from unimodal sources like GPS at these locations look erratic, a thorough analysis of public bus GPS trails for 1,335.365 km at the city of Durgapur, a semi-urban city in India, reveals that several other fine-grained contextual features can characterize these locations accurately. Accordingly, we develop BuStop , a system for extracting and characterizing the stay-locations from multi-modal sensing using commuters’ smartphones. Using this multi-modal information BuStop extracts a set of granular contextual features that allows the system to differentiate among the different stay-location types. A thorough analysis of BuStop using the collected in-house dataset indicates that the system works with high accuracy in identifying different stay-locations such as regular bus stops, random ad hoc stops, stops due to traffic congestion, stops at traffic signals, and stops at sharp turns. Additionally, we develop a proof-of-concept setup on top of BuStop to analyze the potential of the framework in predicting expected arrival time, a critical piece of information required to pre-plan travel at any given bus stop. Subsequent analysis of the PoC framework, through simulation over the test dataset, shows that characterizing the stay-locations indeed helps make more accurate arrival time predictions with deviations less than 60 seconds from the ground-truth arrival time.

Efficient air quality sensing serves as one of the essential services provided in any recent smart city. Mostly facilitated by sparsely deployed Air Quality Monitoring Stations (AQMSs) that are difficult to install and maintain, the overall spatial variation heavily impacts air quality monitoring for locations far enough from these pre-deployed public infrastructures. To mitigate this, we in this article propose a framework named AQuaMoHo that can annotate data obtained from a low-cost thermo-hygrometer (as the sole physical sensing device) with the AQI labels, with the help of additional publicly crawled Spatio-temporal information of that locality. At its core, AQuaMoHo exploits the temporal patterns from a set of readily available spatial features using an LSTM-based model and further enhances the overall quality of the annotation using temporal attention. From a thorough study of two different cities, we observe that AQuaMoHo can significantly help annotate the air quality data on a personal scale.

In this paper, we present the idea of using acoustic sensing over smartphones to understand the internal structure of hollow objects. In the core, we use an elegant, yet lightweight, signal processing pipeline that intelligently uses acoustic chirps to understand the internal structure of the hollow objects. Preliminary experiments on regularly used hollow objects show the potential of the idea.

Multiuser activity recognition has been the core of different context-aware services. However, the development of such services is often plagued by the dearth of multiuser datasets. This paper presents a strategy for generating synthetic multiuser datasets by augmenting existing real-life datasets. The described strategy exploits pre-cise time synchronization and well-known audio augmentation approaches to generate a multimodal activity recognition dataset with locomotive and acoustic signatures.

Severe road accidents are reported regularly across the globe due to drivers getting distracted while using their smartphones. To prevent such fatalities, one possible approach is to make the smartphone intelligent enough to detect whether it is being used by the driver, thus providing restricted access to the applications while driving. However, this problem is challenging as the driver can behave like an adversary to fool the system; therefore, additional devices or forward communication cannot be used. This paper proposes a novel approach of smartphone localization within a car by exploiting the ambient mechanical noise within the vehicle. We utilize the periodic nature of such mechanical noises to develop a simple yet satisfactorily accurate approach, called Blah, that can utilize the acoustic properties from the ambient mechanical noise within the car to detect whether the driver or the passenger is using the smartphone while the car is on the road.

An individual’s cognitive and functional abilities are commonly assessed through physical and mental status examination, observational performance measures, surveys and proxy reports of symptoms. These strategies are not ideal for early impairment detection as the individual needs to be present physically at the clinic to avail the assessments, especially for older adults who require assistance from a caregiver, and experience mobility, cognitive and functional disabilities from neurodegenerative disorders. Moreover, these strategies rely on self-reporting and proxy reports for evaluation which often leads to under-reporting of symptoms and decrease the validity of these measures. We argue that an early assessment of functional, and cognitive health impairment can be obtained from the individual’s daily activities captured through accelerometry. In this work, we postulate to learn high-level motion related representations from accelerometer data to better correlate with underlying functional and cognitive health parameters of older adults using a contrastive and multi-task learning framework. In particular, we posit a novel indicator, Impairment Indicator using the proposed multi-task learning framework that can indicate functional or cognitive decline as neurodegenerative disease progresses. An extensive 24-hour data collection from 25 older adults with the clinician in-the-loop was carried out in a retirement community center with IRB approval. We collected the activity patterns using wearables in their homes in addition to survey-based assessments and observational performance measures recorded by a clinical evaluator to infer their current cognitive and functional impairment status. Our evaluation on the acquired dataset reveals that the representations learned using contrastive learning aids in improving the detection of activities, activity performance score, and stage of dementia to 92%, 97%, and 98%, respectively.

In recent years, non-invasive human activity recognition (HAR) has gathered huge momentum using locomotive sensors. However, for effective HAR, there is a need for a significant volume of annotated data. Typically, the conventional practices for gathering HAR annotations have relied on human annotators. Nevertheless, the growing volume of data often leads to the collection of shallow annotations, which in most cases ignore the fine-grained micro-activities that constitute any complex activities of daily living (ADL). Understanding this, we, in this paper, try to develop the framework AmicroN that can automatically generate micro-activity annotations using locomotive signatures. To achieve this, in the backend, AmicroN applies change-point detection for the precise detection of activity boundaries followed by zero-shot learning with verb attributes to identify the unseen micro-activities without any external supervision. Rigorous evaluation on a publicly available Kitchen dataset shows that AmicroN can identify the micro-activities with a median F1-score of$\geq \mathbf{0.75}$for all the subjects, which can help develop novel pervasive applications.

This demo presents AmicroN which automatically generates fine-grained annotations using locomotive signatures. In the backend, AmicroN exploits short-duration macro labels already present in a pre-annotated dataset. It uses zero-shot learning to identify the finer micro-activities present within a coarse-grain macro-activity label.

Public city bus services across various developing cities inhabit multiple stay-locations on the routes due to ad-hoc bus stops to provide on-demand passenger boarding and alighting services. Characterizing these stay-locations is essential to correctly develop models for bus transit patterns used in various digital navigation services. In this poster, we create a deep learning-driven methodology to characterize ad-hoc stay-locations over bus routes based on crowd-sensing contextual information. Experiments over 720km of bus travel data in a semi-urban city in India indicate promising results from the model in terms of good detection accuracy.

Annotated IMU sensor data from smart devices and wearables are essential for developing supervised models for fine-grained human activity recognition, albeit generating sufficient annotated data for diverse human activities under different environments is challenging. Existing approaches primarily use human-in-the-loop based techniques, including active learning; however, they are tedious, costly, and time-consuming. Leveraging the availability of acoustic data from embedded microphones over the data collection devices, in this paper, we propose LASO, a multimodal approach for automated data annotation from acoustic and locomotive information. LASO works over the edge device itself, ensuring that only the annotated IMU data is collected, discarding the acoustic data from the device itself, hence preserving the audio-privacy of the user. In the absence of any pre-existing labeling information, such an auto-annotation is challenging as the IMU data needs to be sessionized for different time-scaled activities in a completely unsupervised manner. We use a change-point detection technique while synchronizing the locomotive information from the IMU data with the acoustic data, and then use pre-trained audio-based activity recognition models for labeling the IMU data while handling the acoustic noises. LASO efficiently annotates IMU data, without any explicit human intervention, with a mean accuracy of $0.93$ ($\pm 0.04$) and $0.78$ ($\pm 0.05$) for two different real-life datasets from workshop and kitchen environments, respectively.

Most of the latest context-based applications capture the mobility of a user using Inertial Measurement Unit (IMU) sensors like accelerometer and gyroscope which do not need explicit user-permission for application access. Although these sensors provide highly accurate mobility context information, existing studies have shown that they can lead to undesirable leakage of location information. To evade this breach of location privacy, many of the state-of-the-art studies suggest to impose stringent restrictions over the usage of IMU sensors. However, in this paper, we show that typing and smartphone engagement patterns can act as an alternative modality to sniff the mobility context of a user, even if the IMU sensors are not sampled at all. We develop an adversarial framework, named ConType, which exploits the signatures exposed by typing and smartphone engagement patterns to track the mobility of a user. Rigorous experiments with in-the-wild dataset show that ConType can track the mobility contexts with an average micro-F1of 0.87 (±0.09), without using IMU data. Through additional experiments, we also show that ConType can track mobility stealthily with very low power and resource footprints, thus further aggravating the risk.

Internet of Things (IoT) platforms use a large number of low-cost resource constrained devices and generates millions of short-flows. In-network processing is gaining popularity day by day to handle IoT applications and services. However, traditional software-defined networking (SDN) based management systems are not suitable to handle the plug and play nature of such systems. In this paper, we propose Aloe, an auto-scalable SDN orchestration framework. Aloe exploits in-network processing framework by using multiple lightweight controller instances in place of service grade SDN controller applications. The proposed framework ensures the availability and significant reduction in flow-setup delay by deploying instances in the vicinity the resource constraint IoT devices dynamically. Aloe supports fault-tolerance with recovery from network partitioning by employing self-stabilizing placement of migration capable controller instances. Aloe also provides resource reservation for micro-controllers so that they can ensure the quality of services (QoS). The performance of the proposed system is measured by using an in-house testbed along with a large scale deployment in Amazon Web services (AWS) cloud platform. The experimental results from these two testbeds show significant improvement in response time for standard IoT based services. This improvement of performance is due to the reduction in flow-setup time. We found that Aloe can improve flow-setup time by around 10%-30% in comparison to one of the states of the art orchestration framework.

Management of networked Internet of Things (IoT) infrastructure with in-network processing capabilities is becoming increasingly difficult due to the volatility of the system with low-cost resource-constraint devices. Traditional software-defined networking (SDN) based management systems are not suitable to handle the plug and play nature of such systems. Therefore, in this paper, we propose Aloe, an elastically auto-scalable SDN orchestration framework. Instead of using service grade SDN controller applications, Aloe uses multiple lightweight controller instances to exploit the capabilities of in-network processing infrastructure. The proposed framework ensures the availability and significant reduction in flow-setup delay by deploying instances near the resource constraint IoT devices dynamically. Aloe supports fault-tolerance and can recover from network partitioning by employing self-stabilizing placement of migration capable controller instances. The performance of the proposed system is measured by using an in-house testbed along with a large scale deployment in Amazon web services (AWS) cloud platform. The experimental results from these two testbed show significant improvement in response time for standard IoT based services. This improvement of performance is due to the reduction in flow-setup time. We found that Aloe can improve flow-setup time by around 10%-30% in comparison to one of the state of the art orchestration framework.

In an organization, individuals preferto form various formal and informal groups for mutual interactions. Therefore, ubiquitous identification of such groups and understanding their dynamics are important to monitor activities, behaviors, and well-being of the individuals. In this paper, we develop a lightweight, yet near-accurate, methodology, called GroupSense, to identify various interacting groups based on collective sensing through users' smartphones. Group detection from sensor signals is not straightforward because users in proximity may not always be under the same group. Therefore, we use acoustic context extracted from audio signals to infer the interaction pattern among the subjects in proximity. We have developed an unsupervised and lightweight mechanism for user group detection by taking cues from network science and measuring the cohesivity of the detected groups regarding modularity. Taking modularity into consideration, GroupSense can efficiently eliminate incorrect groups, as well as adapt the mechanism depending on the role played by the proximity and the acoustic context in a specific scenario. The proposed method has been implemented and tested under many real-life scenarios in an academic institute environment, and we observe that GroupSense can identify user groups with on an average 0:9(±0:14) F1-Score even in a noisy environment.

In day to day life, people meet strangers while commuting in public transports, roaming around in a shopping mall, waiting at airport boarding areas etc., and thus form passively encountering groups. Detection and analysis of such groups are essential for providing services like targeted advertisements, supply chain management, information broadcasting and so on. However, identifying such groups is challenging because of the underlying dynamics, where an encounter between two subjects is entirely instantaneous without having a specific pattern. This problem has two steps - (a) identification of subjects in proximity and (b) detecting groups from the proximity information. In this paper, we develop an unsupervised model to identify subjects in proximity based on WiFi signal information and assign a proximity score to each pair of subjects based on a novel metric defining the degree of proximity. With the help of these concepts from network science, we then utilize a community detection mechanism to infer the passively encountering groups from the proximity score. The proposed model has been implemented and deployed over an academic institute campus. A study over 25 subjects for six months reveals that the proposed model can detect passively encountering groups with more than 90\% accuracy, even with heterogeneous devices under various real-life scenarios.

Recent context detection techniques in smartphones leverage on the embedded motion sensors, which in turn increases the potential of side-channel attacks. We in this paper propose an alternative modality for obtaining mobility context using smartphone keyboard interaction patterns using a personalized framework. Experimental results show that the framework can predict the mobility context at an average F1 score (both micro and macro) greater than 0.6 across all subjects.

Cross-contamination is incurred during droplet routing in digital microfluidic biochip (DMFB) because of intersection and overlap in the routing paths. Those intersected or overlapped regions are termed as contamination sites. In this paper, we propose a scheme for concurrent routing of multiple droplets with avoidance or minimization in cross-contamination, followed by a wash droplet scheduling operation for residue removal in the contamination sites. The proposed method is based on a hierarchical partitioning method that generates an unbalanced tree structure containing routing nets in the different nodes on the tree. The proposed partitioning technique is guided by the bounding boxes of the routing nets that help in avoiding intersection in routing paths in different partitioned zone/area. During droplet routing, our aim is to optimize resource utilizations, and minimize cross-contamination in a partition. Any cross-contamination site is washed with proper scheduling of wash droplets in the intermediate routing stages, and a shortest path based wash droplet operation is adopted to minimize routing overhead, or total routing completion time. Different observations are made by applying our algorithm on real benchmarks, and experimental results show improvement in many cases.