Heterogeneous sensors (e.g., 2D cameras and LiDAR) provide a novel opportunity to leverage multiple modalities in collaborative artificial intelligence (AI)-based video analytics pipelines. Such applications use sensors that are frequently attached to resource-limited edge devices which can hinder the execution of multimodal and deep DNN models. While powerful edge devices can still benefit from multimodal fusion to enhance robustness, joint training of such models for generalizable applications is often infeasible due to the lack of large-scale multimodal datasets and the prohibitive cost involved in annotating those datasets. To address this, we introduce FusionBridge: a lightweight fusion framework that combines the capabilities of independently trained 2D (image-based) and 3D (LiDAR-based) perception models to improve object detection at the edge. FusionBridge extracts mid-level features from single modality 3D models and performs cross-modal fusion via a lightweight transformer-based adapter. This enables hints to be exchanged without requiring joint end-to-end training. By bridging modality-specific experts, our approach maintains modularity, supports model reuse, and allows scalable deployment across heterogeneous sensor configurations with zero calibration or sensor alignment effort. Evaluations on simulated and real world deployments demonstrate that FusionBridge achieves up to a 57% F1-score improvement over any single-modality baseline, while only incurring a 15% latency overhead and 0.4KB/frame transmission overhead compared to the baseline.

Abstract Image memorability (IM) estimation typically relies on learning generic semantic features from large-scale datasets; however, memorability is intrinsically individual-dependent and personalized visual viewing behavior, but overlooking this variability can undermine model performance in downstream cognitive and human–machine interaction tasks, leading toward suboptimal performance. To address this, we propose PerMem , a unified framework that integrates knowledge distillation (KD) and imitation learning (IL) to jointly learn generic and personalized salient representations for memorability estimation by leveraging both image content and gaze-derived heatmaps. PerMem employs a teacher network built upon a pre-trained ResNet-50 backbone, followed by an encoder–decoder architecture coupled with spatial and channel attention mechanisms to generate generic saliency-aware memorability maps. A lightweight, attention-guided student encoder–decoder is then optimized through a composite imitation-guided distillation process, where knowledge is distilled from the teacher while simultaneously imitating user-specific gaze fixation heatmaps. Through this joint training process, the student network learns to produce personalized memorability estimates while achieving substantial reductions in computational complexity. We validate PerMem on two public IM estimation datasets (LaMem and SUN) and a in-house WoM dataset comprising of 45 participants (UMBC IRB #670) engaged in visual search and navigation tasks that reflect individualized visual attention patterns. PerMem outperforms nine state-of-the-art IM models by capturing coarse-to-fine saliency and adapting to individual attention, achieving $$\approx $$ ≈ 6% improvement in memorability prediction. Lastly, we evaluate PerMem on heterogeneous embedded edge devices, including Jetson Nano, Jetson Xavier NX, and Raspberry Pi, demonstrating efficient on-device inference with consistent reductions in memory usage (25.6–33.4%), power consumption (19.0–25.0%), and inference latency (34.9–39.8%) relative to the teacher model, highlighting its practical robustness for resource-constrained, human-centric applications.

Image memorability (IM) estimation typically relies on learning generic semantic features from large-scale datasets; however, memorability is intrinsically individual-dependent and personalized visual viewing behavior, but overlooking this variability can undermine model performance in downstream cognitive and human-machine interaction tasks, leading towards suboptimal performance. To address this, we propose MemGaze, a unified framework that integrates knowledge distillation (KD) and imitation learning (IL) to jointly learn generic and personalized salient representations for memorability estimation by leveraging both image content and gaze-derived heatmaps. MemGaze employs a teacher network built upon a pretrained ResNet-50 backbone, followed by an encoder-decoder architecture coupled with spatial and channel attention mechanisms to generate generic saliency-aware memorability maps. A lightweight, attention-guided student encoder-decoder is then optimized through a composite imitation-guided distillation process, where knowledge is distilled from the teacher while simultaneously imitating user-specific gaze fixation heatmaps. Through this joint training process, the student network learns to produce personalized memorability estimates while achieving substantial reductions in computational complexity. We validate MemGaze on two public IM estimation datasets (LaMem and SUN) and a inhouse WoM dataset comprising of 45 participants (UMBC IRB #670) engaged in visual search and navigation tasks that reflect individualized visual attention patterns. MemGaze outperforms nine state-of-the-art IM models by capturing coarse-to-fine saliency and adapting to individual attention, achieving ≈6% improvement in memorability prediction.

ACM UMAP June 16-19 2025, New York, USA

In this paper, we propose GestRight, a real-time system for gesture-based tele-operation of a mobile robot. For field use (e.g., smart factory settings, search and rescue missions, etc.), relying on tablet-based controls or joysticks are limiting which has led to the recent interest in hands-free operation of these assistive robots. In this work, we design three gesture-based schemes, namely, fist, touch, and wheel, represent three levels of precision–intuitiveness tradeoffs for low-level navigational control of mobile robots. GestRight includes a head-mounted device that captures hand joint data for accurate gesture recognition which is then translated to motion commands at an edge server. Through a user study involving seventeen participants, we present quantitative insights in comparison to traditional modes of control. Specifically, we evaluate GestRight in terms of the ease of navigational control, task time, and amount of errors/corrective actions required, run extensive statistical analyses, and provide a series of design recommendations for gesture-driven teleoperation systems. Our results show that gesture based schemes perform as well as traditional modes of control in contrast to participants’ self-reports on how successful they felt in controlling the robots.

Electroencephalography (EEG) has found many applications cutting across many domains, such as digital health, affective computing, and human-machine interfaces. However, its widespread adoption in practice has been primarily inhibited by its susceptibility to noise artifacts and the low spatial resolution of electrodes on commercial EEG sensors. While several prior works have investigated techniques for detecting and extracting noise, our understanding of performance degradation due to electrode sparsity remains limited. In this work, we explore the feasibility of using Functional Connectivity (FC) for improving the EEG sensing-based accuracy using two exemplars downstream, working memory-related tasks: (a) cognitive task load (CTL) assessment and (b) high attentional event-evoked potential (EEP) episodes detection. This paper proposes an integrated approach, EEGAmp+ , that first utilizes channel-wise functional connectivity modules using independent component analysis (ICA) coupled with cosine distance for EEG signal reconstruction for cognitive task load assessment tasks. This is then coupled with a sliding window change point detection technique paired with continuous wavelet transformation (CWT) to extract high attentional EEP episodes. Our empirical results indicate that using independent component analysis (ICA) coupled with FC to improve spatial resolution increased cognitive load assessment accuracy by [5.6% ± 1.13] across four machine learning algorithms. Furthermore, after signal reconstruction, we introduce sliding window CPD coupled with CWT, which allows us to extract EEP segments legibly through decomposing the signals and the ability to capture both time and frequency representation from the reconstructed signal boosting detection accuracy by [11.1% ±1.31].

While Deep Neural Network (DNN) models have transformed machine vision capabilities, their extremely high computational complexity and model sizes present a formidable deployment roadblock for AIoT applications. We show that the complexity-vs-accuracy-vs-communication tradeoffs for such DNN models can be significantly addressed via a novel, lightweight form of "collaborative machine intelligence" that requires only runtime changes to the inference process. In our proposed approach, called ComAI, the DNN pipelines of different vision sensors share intermediate processing state with one another, effectively providing hints about objects located within their mutually-overlapping Field-of-Views (FoVs). CoMAI uses two novel techniques: (a) a secondary shallow ML model that uses features from early layers of a peer DNN to predict object confidence values in the image, and (b) a pipelined sharing of such confidence values, by collaborators, that is then used to bias a reference DNN’s outputs. We demonstrate that CoMAI (a) can boost accuracy (recall) of DNN inference by 20-50%, (b) works across heterogeneous DNN models and deployments, and (c) incurs negligible processing, bandwidth and processing overheads compared to non-collaborative baselines.

The proliferation of connected embedded devices, or the Internet of Things (IoT), together with recent advances in machine intelligence, will change the profile of future cloud services and introduce a variety of new research problems, both in cloud applications and infrastructure layers. These problems are centered around empowering individually resource-limited devices to exhibit intelligent behavior, both in sensing and control, thanks to a judicious utilization of cloud resources. Cloud services will enable learning from data, perform inference, and execute control, all with assurances on outcomes. This article discusses such emerging services and outlines five resulting new research directions towards enabling and optimizing intelligent, cloud-assisted sensing and control in the age of the Internet of Things.

The paper discusses an emerging suite of machine intelligence services that are of increasing importance in the highly instrumented world of the Internet of Things (IoT). The suite, called Eugene, would offer a form of intelligent behavior (based on deep neural networks) to otherwise simple embedded devices; the clients of the service. These devices would benefit from service resources to learn from data and to perform intelligent inference, classification, prediction, and estimation tasks that they are too limited to carry out on their own. The paper discusses the taxonomy of such services and the state of implementation, as well as the various challenges entailed, including scheduling, caching (of intelligent functions), and cooperative learning.

While analysis of urban commuting data has a long and demonstrated history of providing useful insights into human mobility behavior, such analysis has been performed largely in offline fashion and to aid medium-to-long term urban planning. In this work, we demonstrate the power of applying predictive analytics on real-time mobility data, specifically the smart-card generated trip data of millions of public bus commuters in Singapore, to create two novel and "live" smart city services. The key analytical novelty in our work lies in combining two aspects of urban mobility: (a) conformity: which reflects the predictability in the aggregated flow of commuters along bus routes, and (b) regularity: which captures the repeated trip patterns of each individual commuter. We demonstrate that the fusion of these two measures of behavior can be performed at city-scale using our BuScope platform, and can be used to create two innovative smart city applications. The Last-Mile Demand Generator provides O(mins) lookahead into the number of disembarking passengers at neighborhood bus stops; it achieves over 85% accuracy in predicting such disembarkations by an ingenious combination of individual-level regularity with aggregate-level conformity. By moving driverless vehicles proactively to match this predicted demand, we can reduce wait times for disembarking passengers by over 75%. Independently, the Neighborhood Event Detector uses outlier measures of currently operating buses to detect and spatiotemporally localize dynamic urban events, as much as 1.5 hours in advance, with a localization error of ~450 meters.

In this paper, we present LiveLabs, a first-of-its-kind testbed that is deployed across a university campus, convention centre, and resort island and collects real-time attributes such as location, group context etc., from hundreds of opt-in participants. These venues, data, and participants are then made available for running rich human-centric behavioural experiments that could test new mobile sensing infrastructure, applications, analytics, or more social-science type hypotheses that influence and then observe actual user behaviour. We share case studies of how researchers from around the world have and are using LiveLabs, and our experiences and lessons learned from building, maintaining, and expanding Live-Labs over the last three years.

This paper investigates how digital traces of people's movements and activities in the physical world (e.g., at college campuses and commutes) may be used to detect local, short-lived events in various urban spaces. Past work that use occupancy-related features can only identify high-intensity events (those that cause large-scale disruption in visit patterns). In this paper, we first show how longitudinal traces of the coordinated and group-based movement episodes obtained from individual-level movement data can be used to create a socio-physical network (with edges representing tie strengths among individuals based on their physical world movement & collocation behavior). We then investigate how two additional families of socio-physical features: (i) group-level interactions observed over shorter timescales and (ii) socio-physical network tie-strengths derived over longer timescales, can be used by state-of-the-art anomaly detection methods to detect a much wider set of both high & low intensity events. We utilize two distinct datasets--one capturing coarse-grained SMU campus-wide indoor location data from hundreds of students, and the other capturing commuting behavior by millions of users on Singapore's public transport network--to demonstrate the promise of our approaches: the addition of group and socio-physical tie-strength based features increases recall (the percentage of events detected) more than 2-folds (to 0.77 on the SMU campus and to 0.73 at sample MRT stations), compared to pure occupancy-based approaches.

In this paper, we show that characterizing user behaviour from location or smartphone usage traces, without accounting for the interaction of individuals in physical-world groups, can lead to erroneous results. We conducted one of the largest studies in the UbiComp domain thus far, involving indoor location traces of more than 6,000 users, collected over a 4-month period at our university campus, and further studied fine-grained App usage of a subset of 156 Android users. We apply a state-of-the-art group detection algorithm to annotate such location traces with group vs. individual context, and then show that individuals vs. groups exhibit significant differences along three behavioural traits: (1) the mobility pattern, (2) the responsiveness to calls / SMSs and (3) application usage. We show that these significant differences are robust to underlying errors in the group detection technique and that the use of such group context leads to behavioural results that differ from those reported in prior popular work.

The US Army Research Laboratory (ARL) and the Air Force Research Laboratory (AFRL) have established a collaborative research enterprise referred to as the Situational Understanding Research Institute (SURI). The goal is to develop an information processing framework to help the military obtain real-time situational awareness of physical events by harnessing the combined power of multiple sensing sources to obtain insights about events and their evolution. It is envisioned that one could use such information to predict behaviors of groups, be they local transient groups (e.g., Protests) or widespread, networked groups, and thus enable proactive prevention of nefarious activities. This paper presents a vision of how social media sources can be exploited in the above context to obtain insights about events, groups, and their evolution.

Detecting the group context of an individual (i.e., whether an individual is alone or part of a group) in crowded public spaces, such as shopping malls, is an important goal with many practical applications. However, in crowded indoor spaces, understanding the group-dependent movement behavior is a non-trivial problem as: (1) detecting groups is hard as the density ensures that at any location, a large number of people are moving together, (2) location tracking in many real-world venues is either absent or not very accurate, and (3) indoor mobility models that take into account group attributes (such as group size) are rare. In this paper, we first introduce GruMon, a platform for near real-time group monitoring in dense, public spaces, and then demonstrate how the movement & residency properties of individuals are significantly affected when they are in groups.

Real-time monitoring of groups and their rich contexts will be a key building block for futuristic, group-aware mobile services. In this paper, we propose GruMon, a fast and accurate group monitoring system for dense and complex urban spaces. GruMon meets the performance criteria of precise group detection at low latencies by overcoming two critical challenges of practical urban spaces, namely (a) the high density of crowds, and (b) the imprecise location information available indoors. Using a host of novel features extracted from commodity smartphone sensors, GruMon can detect over 80% of the groups, with 97% precision, using 10 minutes latency windows, even in venues with limited or no location information. Moreover, in venues where location information is available, GruMon improves the detection latency by up to 20% using semantic information and additional sensors to complement traditional spatio-temporal clustering approaches. We evaluated GruMon on data collected from 258 shopping episodes from 154 real participants, in two large shopping complexes in Korea and Singapore. We also tested GruMon on a large-scale dataset from an international airport (containing ≈37K+ unlabelled location traces per day) and a live deployment at our university, and showed both GruMon's potential performance at scale and various scalability challenges for real-world dense environment deployments.

Existing algorithms for trajectory-based clustering usually rely on simplex representation and a single proximity-related distance (or similarity) measure. Consequently, additional information markers (e.g., social interactions or the semantics of the spatial layout) are usually ignored, leading to the inability to fully discover the communities in the trajectory database. This is especially true for human-generated trajectories, where additional fine-grained markers (e.g., movement velocity at certain locations, or the sequence of semantic spaces visited) can help capture latent relationships between cluster members. To address this limitation, we propose TODMIS: a general framework for Trajectory cOmmunity Discovery using Multiple Information Sources. TODMIS combines additional information with raw trajectory data and creates multiple similarity metrics. In our proposed approach, we first develop a novel approach for computing semantic level similarity by constructing a Markov Random Walk model from the semantically-labeled trajectory data, and then measuring similarity at the distribution level. In addition, we also extract and compute pair-wise similarity measures related to three additional markers, namely trajectory level spatial alignment (proximity), temporal patterns and multi-scale velocity statistics. Finally, after creating a single similarity metric from the weighted combination of these multiple measures, we apply dense sub-graph detection to discover the set of distinct communities. We evaluated TODMIS extensively using traces of (i) student movement data in a campus, (ii) customer trajectories in a shopping mall, and (iii) city-scale taxi movement data. Experimental results demonstrate that TODMIS correctly and efficiently discovers the real grouping behaviors in these diverse settings.

Advertisements are the de-facto currency of the Internet with many popular applications (e.g. Angry Birds) and online services (e.g., YouTube) relying on advertisement generated revenue. However, the current economic models and mechanisms for mobile advertising are fundamentally not sustainable and far from ideal. In particular, as we show, applications which use mobile advertising are capable of using significant amounts of a mobile users' critical resources without being controlled or held accountable. This paper seeks to redress this situation by enabling advertisement supported applications to become significantly more ``user-friendly''. To this end, we present the design and implementation of CAMEO, a new framework for mobile advertising that 1) employs intelligent and proactive retrieval of advertisements, using context prediction, to significantly reduce the bandwidth and energy overheads of advertising, and 2) provides a negotiation protocol and framework that empowers applications to subsidize their data traffic costs by ``bartering'' their advertisement rights for access bandwidth from mobile ISPs. Our evaluation, that uses real mobile advertising data collected from around the globe, demonstrates that CAMEO effectively reduces the resource consumption caused by mobile advertising.