Multimodal human action recognition (HAR) utilizes complementary data for activity classification. Built on traditional HAR tasks, recent advances in Large Language Models (LLMs) enable detailed descriptions and causal reasoning of human actions, advancing new tasks of human action understanding (HAU) and human action reasoning (HARn). However, most LLMs, especially multimodal Large Vision-Language Models (LVLMs), struggle with modalities other than RGB images, like depth, IMU, ormmWave, due to a lack of large-scale datasets in these task domains. Existing HAR datasets provide only coarse-grained annotations, in-sufficient for depicting the detailed action dynamics required in HAU and HARn tasks. Simply combining annotations and generating captions with LLMs often lacks necessary logical and spatiotemporal consistency. In this paper, we introduce CUHK-X, a large-scale multi-modal dataset and benchmarks for HAR, HAU, and HARn. It includes 64,267 samples of 40 actions performed by 30 participants across two indoor environments, covering diverse daily scenarios. To address the challenge of spatiotemporal inconsistencies in captions, we propose a prompt-based scene creation method that leverages LLMs to generate logically connected activity sequences. CUHK-X also includes three benchmarks with six tasks to evaluate state-of-the-art models. Experimental results show average accuracies of 76.52% for HAR, 40.76% for HAU, and 70.25% for HARn. This large-scale multimodal dataset aims to empower the research community to apply, develop, and adapt data-intensive learning techniques for a wide range of human activity-related tasks.

Alzheimer's Disease (AD) and related dementia are a growing global health challenge due to the aging population. In this paper, we present ADMarker, the first end-to-end system that integrates multi-modal sensors and new federated learning algorithms for detecting multidimensional AD digital biomarkers in natural living environments. ADMarker features a novel three-stage multi-modal federated learning architecture that can accurately detect digital biomarkers in a privacy-preserving manner. Our approach collectively addresses several major real-world challenges, such as limited data labels, data heterogeneity, and limited computing resources. We built a compact multi-modality hardware system and deployed it in a four-week clinical trial involving 91 elderly participants. The results indicate that ADMarker can accurately detect a comprehensive set of digital biomarkers with up to 93.8% accuracy and identify early AD with an average of 88.9% accuracy. ADMarker offers a new platform that can allow AD clinicians to characterize and track the complex correlation between multidimensional interpretable digital biomarkers, demographic factors of patients, and AD diagnosis in a longitudinal manner.

Alzheimer's disease and related dementia has emerged as a global health challenge due to aging population. Expressed Emotion (EE) is a widely-used medical measure of family emotional environment of patients with caregivers. We present EmoMarker, a multi-modal sensor detection system for dyadic digital biomarkers of EE in dementia patients' homes. EmoMarker consists of a privacy-preserving depth camera and a microphone to extracts interpretable dyadic (i.e., motor and acoustic) digital biomarkers of the interaction between patients and caregivers and predict the scores of Family Altitude Scale, an assessment tool for measuring the emotional climate of families. We have deployed our system in 99 elder people's homes and achieved 81.13% prediction accuracy in preliminary results.

This paper presents TouchAuth, a new touch-to-access device authentication approach using induced body electric potentials (iBEPs) caused by the indoor ambient electric field that is mainly emitted from the building's electrical cabling. The design of TouchAuth is based on the electrostatics of iBEP generation and a resulting property, i.e., the iBEPs at two close locations on the same human body are similar, whereas those from different human bodies are distinct. Extensive experiments verify the above property and show that TouchAuth achieves high-profile receiver operating characteristics in implementing the touch-to-access policy. Our experiments also show that a range of possible interfering sources including appliances' electromagnetic emanations and noise injections into the power network do not affect the performance of TouchAuth. A key advantage of TouchAuth is that the iBEP sensing requires a simple analog-to-digital converter only, which is widely available on microcontrollers. Compared with existing approaches including intra-body communication and physiological sensing, TouchAuth is a low-cost, lightweight, and convenient approach for authorized users to access the smart objects found in indoor environments.

Design of clock synchronization for networked nodes faces a fundamental trade-off between synchronization accuracy and universality of heterogeneous platforms, because a high synchronization accuracy generally requires platform-dependent hardware-level network packet timestamping. This paper presents TouchSync, a new indoor clock synchronization approach for wearables that achieves millisecond accuracy while preserving universality in that it uses standard system calls only, such as reading system clock, sampling sensors, and sending/receiving network messages. The design of TouchSync is driven by a key finding from our extensive measurements that the skin electric potentials (SEPs) induced by powerline radiation are salient, periodic, and synchronous on a same wearer and even across different wearers. TouchSync integrates the SEP signal into the universal principle of Network Time Protocol and solves an integer ambiguity problem by fusing the ambiguous results in multiple synchronization rounds to conclude an accurate clock offset between two synchronizing wearables. With our shared code, TouchSync can be readily integrated into any wearable applications. Extensive evaluation based on our Arduino and TinyOS implementations shows that TouchSync's synchronization errors are below 3 and 7 milliseconds on the same wearer and between two wearers 10 kilometers apart, respectively.

Various sensing modalities have been exploited for indoor location sensing, each of which has well understood limitations, however. This paper presents a first systematic study on using the electromagnetic field (EMF) induced by a building's electric power network for simultaneous localization and mapping (SLAM). A basis of this work is a measurement study showing that the power network EMF sensed by either a customized sensor or smartphone's microphone as a side-channel sensor is spatially distinct and temporally stable. Based on this, we design a SLAM approach that can reliably detect loop closures based on EMF sensing results. With the EMF feature map constructed by SLAM, we also design an efficient online localization scheme for resource-constrained mobiles. Evaluation in three indoor spaces shows that the power network EMF is a promising modality for location sensing on mobile devices, which is able to run in real time and achieve sub-meter accuracy.

The continuous fluctuation of electric network frequency (ENF) presents a fingerprint indicative of time, which we call natural timestamp . This article studies the time accuracy of these natural timestamps obtained from powerline electromagnetic radiation (EMR), which is mainly excited by powerline voltage oscillations at the rate of the ENF. However, since the EMR signal is often weak and noisy, extracting the ENF is challenging, especially on resource-limited sensor platforms. We design an efficient EMR conditioning algorithm and evaluate the time accuracy of EMR natural timestamps on two representative classes of IoT platforms—a high-end single-board computer with a customized EMR antenna and a low-end mote with a normal conductor wire acting as EMR antenna. Extensive measurements at six sites in a city, which are away from each other for up to 24km, show that the high-end and low-end nodes achieve median time errors of about 50ms and 150ms, respectively. To demonstrate the use of the EMR natural timestamps, we discuss three applications: time recovery, runtime clock verification, and secure clock synchronization.

The continuous fluctuation of electric network frequency (ENF) presents a fingerprint indicative of time, which we call natural timestamp. This paper studies the time accuracy of these natural timestamps obtained from powerline electromagnetic radiation (EMR), which is mainly excited by powerline voltage oscillations at the rate of the ENF. However, since the EMR signal is often weak and noisy, extracting the ENF is challenging, especially on resource-limited sensor platforms. We design an efficient EMR conditioning algorithm and evaluate the time accuracy of EMR natural timestamps on two representative classes of IoT platforms - a high-end single-board computer with a customized EMR antenna and a low-end mote with a normal conductor wire acting as EMR antenna. Extensive measurements at five sites in a city, which are away from each other for up to 24 km, show that the high-end and low-end nodes achieve median time errors of about 50 ms and 150 ms, respectively. To demonstrate the use of the EMR natural timestamps, we discuss two applications, namely time recovery and runtime clock verification.

The continuous fluctuation of electric network frequency (ENF) presents a fingerprint indicative of time, which we call natural timestamp. This live demo demonstrates the accuracy of the natural timestamps obtained by four wired voltage sensors and four wireless electromagnetic radiation (EMR) sensors that are geographically distributed in Singapore. The voltage sensors and the EMR sensors capture the minute fluctuations of the length of each voltage cycle and the average ENF over every 50 voltage cycles, respectively. The evaluation in our prior studies [1, 3] has shown that the natural timestamps recorded by the voltage sensors and the EMR sensors give sub-millisecond and sub-second average time errors, respectively. This demo will also show their time errors.

Design of clock synchronization for networked nodes faces a fundamental trade-off between synchronization accuracy and universality for heterogeneous platforms, because a high synchronization accuracy generally requires platform-dependent hardware-level network packet timestamping. This paper presents TouchSync, a new indoor clock synchronization approach for wearables that achieves millisecond accuracy while preserving universality in that it uses standard system calls only, such as reading system clock, sampling sensors, and sending/receiving network messages. The design of TouchSync is driven by a key finding from our extensive measurements that the skin electric potentials (SEPs) induced by powerline radiation are salient, periodic, and synchronous on a same wearer and even across different wearers. TouchSync integrates the SEP signal into the universal principle of Network Time Protocol and solves an integer ambiguity problem by fusing the ambiguous results in multiple synchronization rounds to conclude an accurate clock offset between two synchronizing wearables. With our shared code, TouchSync can be readily integrated into any wearable applications. Extensive evaluation based on our Arduino and TinyOS implementations shows that TouchSync's synchronization errors are below 3 and 7 milliseconds on the same wearer and between two wearers 10 kilometers apart, respectively.