Hyperdimensional computing (HDC) is an approach from the cognitive science literature for solving information processing tasks using data represented as high-dimensional random vectors. The technique has a rigorous mathematical backing, and is easy to implement in energy-efficient and highly parallel hardware like FPGAs and "processing-in-memory" architectures. The effectiveness of HDC in machine learning largely depends on how raw data is mapped to high-dimensional space. In this work, we propose NysHD, a new method for constructing this mapping that is based on the Nyström method from the literature on kernel approximation. Our approach provides a simple recipe to turn any user-defined positive-semidefinite similarity function into an equivalent mapping in HDC. There is a vast literature on the design of such functions for learning problems. Our approach provides a mechanism to import them into the HDC setting, expanding the types of problems that can be tackled using HDC. Empirical evaluation against existing HDC encoding methods shows that NysHD can achieve, on average, 11% and 17% better classification accuracy on graph and string datasets respectively.

Rich and context-aware activity logs facilitate user behavior analysis and health monitoring, making them a key research focus in ubiquitous computing. The remarkable semantic understanding and generation capabilities of Large Language Models (LLMs) have recently created new opportunities for activity log generation. However, existing methods continue to exhibit notable limitations in terms of accuracy, efficiency, and semantic richness. To address these challenges, we propose DailyLLM. To the best of our knowledge, this is the first log generation and summarization system that comprehensively integrates contextual activity information across four dimensions: location, motion, environment, and physiology, using only sensors commonly available on smartphones and smartwatches. To achieve this, DailyLLM introduces a lightweight LLM-based framework that integrates structured prompting with efficient feature extraction to enable high-level activity understanding. Extensive experiments demonstrate that DailyLLM outperforms state-of-the-art (SOTA) log generation methods and can be efficiently deployed on personal computers and Raspberry Pi. Utilizing only a 1.5B-parameter LLM model, DailyLLM achieves a 17% improvement in log generation BERTScore precision compared to the 70B-parameter SOTA baseline, while delivering nearly 10× faster inference speed.

With the rapid growth in sensor data, effectively interpreting and interfacing with these data in a human-understandable way has become crucial. While existing research primarily focuses on learning classification models, fewer studies have explored how end users can actively extract useful insights from sensor data, often hindered by the lack of a proper dataset. To address this gap, we introduce SensorQA, the first human-created question-answering (QA) dataset for daily life monitoring, based on long-term time-series sensor data. SensorQA is created by human workers and includes 5.6K diverse and practical queries that reflect genuine human interests, paired with accurate answers derived from the sensor data. We further establish benchmarks for state-of-the-art AI models on this dataset and evaluate their performance on typical edge devices. Our results reveal a gap between current models and optimal QA performance as well as efficiency, highlighting the need for new contributions. The dataset and code are available at: https://github.com/benjamin-reichman/SensorQA.

Detailed activity logs are crucial for health monitoring and personalized interventions. Traditional methods rely on manual editing or raise privacy concerns due to the use of camera recordings. This paper proposes ContextLLM, an innovative system that utilizes a large language model (LLM) to understand sensor data from smartphones and smartwatches and automatically generate contextualized activity logs. Compared to the state-of-the-art, it incorporates key contextual information and physiological indicators, enabling more fine-grained semantic descriptions. Preliminary results show that the automatically generated activity logs achieve 80.26% similarity to human annotations, demonstrating the feasibility.

Federated learning is a distributed learning method by training the model in locally multiple clients, which has been used in numerous fields. Current convolutional neural networks (CNN)-based federated learning approaches face challenges from computational cost, communication efficiency, and robust communication. Recently, Hyper Dimensional Computing (HDC) has been recognized as a promising technique to address these challenges. HDC encodes data as high-dimensional vectors and enables lightweight training and communication through simple parallel vector operations. Several HDC-based federated learning methods have been proposed. Although existing methods reduce computational efficiency and communication cost, they are difficult to handle complex learning tasks and are not robust to unreliable wireless channels. In this work, we innovatively introduce a synergetic federated learning framework, FHDnn. With advantage of the complementary strengths of CNN and HDC, FHDnn can achieve optimal performance on complex image tasks while maintaining good computational and communication efficiency. Secondly, we demonstrate in detail the convergence of using HDC in a generalized federated learning framework, providing theoretical guarantees for HDC-based federated learning approach. Finally, we design three communication strategies to further improve the communication efficiency of FHDnn by 32×. Experiments demonstrate that FHDnn converges 3× faster than CNN-based federated learning methods, reduces the communication cost by 2,112×, and the local computation and energy consumption by 192×. In addition, it has good robustness to unreliable communication with bit errors, noise, and packet loss.

Time series forecasting is shifting towards Edge AI, where models are trained and executed on edge devices instead of in the cloud. However, training forecasting models at the edge faces two challenges concurrently: (1) dealing with streaming data containing abundant noise, which can lead to degradation in model predictions, and (2) coping with limited on-device resources. Traditional approaches focus on simple statistical methods like ARIMA or neural networks, which are either not robust to sensor noise or not efficient for edge deployment, or both. In this paper, we propose a novel, robust, and lightweight method named KalmanHD for on-device time series forecasting using Hyperdimensional Computing (HDC). KalmanHD integrates Kalman Filter (KF) with HDC, resulting in a new regression method that combines the robustness of KF towards sensor noise and the efficiency of HDC. KalmanHD first encodes the past values into a high-dimensional vector representation, then applies the Expectation-Maximization (EM) approach as in KF to iteratively update the model based on the incoming samples. KalmanHD inherently considers the variability of each sample and thereby enhances robustness. We further accelerate KalmanHD by substituting the expensive matrix multiplication with efficient binary operations between the covariance and the encoded values. Our results show that KalmanHD achieves MAE comparable to the state-of-the-art noise-optimized NN-based methods while running $3.6-8.6\times$ faster on typical edge platforms. The source code is available at https://github.com/DarthIV02/Ka1manHD

Federated Learning (FL) has gained increasing interest as a privacy-preserving distributed learning paradigm in recent years. Although previous works have addressed data and system heterogeneities in FL, there has been less exploration of modality heterogeneity, where clients collect data from various sensor types such as accelerometer, gyroscope, etc. As a result, traditional FL methods assuming uni-modal sensors are not applicable in multimodal federated learning (MFL). State-of-the-art MFL methods use modality-specific blocks, usually recurrent neural networks, to process each modality. However, executing these methods on edge devices proves challenging and resource-intensive. A new MFL algorithm is needed to jointly learn from heterogeneous sensor modalities while operating within limited resources and energy. We propose a novel hybrid framework based on Hyperdimensional Computing (HD) and deep learning, named MultimodalHD, to learn effectively and efficiently from edge devices with different sensor modalities. MultimodalHD uses a static HD encoder to encode raw sensory data from different modalities into high-dimensional low-precision hypervectors. These multimodal hypervectors are then fed to an attentive fusion module for learning richer representations via inter-modality attention. Moreover, we design a proximity-based aggregation strategy to alleviate modality interference between clients. MultimodalHD is designed to fully utilize the strengths of both worlds: the computing efficiency of HD and the capability of deep learning. We conduct experiments on multimodal human activity recognition datasets. Results show that MultimodalHD delivers comparable (if not better) accuracy compared to state-of-the-art MFL algorithms, while being 2x – 8x more efficient in terms of training time. Our code is available online1.

On-device learning has emerged as a prevailing trend that avoids the slow response time and costly communication of cloud-based learning. The ability to learn continuously and indefinitely in a changing environment, and with resource constraints, is critical for real sensor deployments. However, existing designs are inadequate for practical scenarios with (i) streaming data input, (ii) lack of supervision and (iii) limited on-board resources. In this paper, we design and deploy the first on-device lifelong learning system called LifeHD for general IoT applications with limited supervision. LifeHD is designed based on a novel neurally-inspired and lightweight learning paradigm called Hyperdimensional Computing (HDC). We utilize a two-tier associative memory organization to intelligently store and manage high-dimensional, low-precision vectors, which represent the historical patterns as cluster centroids. We additionally propose two variants of LifeHD to cope with scarce labeled inputs and power constraints. We implement LifeHD on off-the-shelf edge platforms and perform extensive evaluations across three scenarios. Our measurements show that LifeHD improves the unsupervised clustering accuracy by up to 74.8% compared to the state-of-the-art NN-based unsupervised lifelong learning baselines with as much as 34.3x better energy efficiency. Our code is available at https://github.com/Orienfish/LifeHD.

On-device environmental sound classification (ESC) in rural areas faces one major challenge of resource efficiency. Traditional methods rely on resource-intensive machine learning models, making them impractical for small edge devices like microcontrollers (MCUs). This poster presents SoundHD, a novel ESC solution using Hyperdimensional Computing (HDC), a brain-inspired and lightweight computing paradigm. We further optimize the memory footprint for deployment on MCUs. Our initial results show that SoundHD can be deployed and executed effectively on memory-constrained MCUs.

Question Answering (QA) establishes a natural and intuitive way for humans to interpret and understand multimodal sensor data. However, existing sensor-based QA systems are limited in the types of questions & answers, and the duration of sensor data they can handle. In this demo, we introduce an end-to-end QA system for long-term multimodal timeseries sensors powered by Large Language Models (LLMs). Our system features a novel pipeline with LLM-based question decomposition, sensor data query and LLM-based answer assembly. We further quantize the LLMs and deploy our system on two typical edge platforms, delivering higher-quality answers with low latency.

Recent years have seen significant progress in unsupervised continual learning methods. Despite their success in controlled settings, their practicality in real-world contexts remains uncertain. In this paper, we first empirically investigate existing self-supervised continual learning methods. We show that even with a replay buffer, existing methods cannot preserve the critical knowledge on videos with temporal-correlated input. Our insight is that the primary challenge of unsupervised continual learning stems from the unpredictable input and the absence of supervision as well as prior knowledge. Drawing inspiration from hybrid AI, we introduce Evolve, an innovative framework employing multiple pretrained models in the cloud, as experts, to bolster existing self-supervised learning methods on local clients. Evolve harnesses expert guidance through a novel expert aggregation loss, calculated and returned from the cloud. It also dynamically assigns weights to experts based on their confidence and tailored prior knowledge, thereby offering adaptive supervision for new streaming data. We extensively validate Evolve across several real-world data streams with temporal correlation. The results convincingly demonstrate that Evolve surpasses the best state-of-the-art unsupervised continual learning method by 6.1-53.7% in top-1 linear evaluation accuracy across various data streams, affirming the efficacy of diverse expert guidance. The codebase is at https://github.com/Orienfish/Evolve.

As a lightweight and robust brain-inspired computing paradigm, Hyperdimensional Computing (HDC) serves as a promising solution for the next-generation edge AI. However, the basic form of HDC is vulnerable to privacy leaks and cyber attacks. In this paper, we breifly review and discuss the recent contributions to privacy and security of HDC. We first summarize existing HDC designs to protect against privacy leaks, such as differential privacy. Next, we review the data encryption techniques for collaborative learning using HDC based on Multi-Party Computation and Homomorphic Encryption. Finally, we discuss the HDC-based designs for combating cyber attacks in a malicious environment. More research on private and secure HDC-based methods are needed for future large-scale edge deployment.

Along with the recent advancements of lightweight machine learning and powerful systems and hardware platforms, intelligence beyond the edge has become the next tide of IoT. However, multiple barriers exist from data, algorithm, network and hardware perspectives. In this abstract, I provide an overview of my PhD research which aims at closing the gap towards deploying edge intelligence for large-scale and real-world IoT applications. I further introduce our recent contributions and the work planned ahead.

With the continuing advancement of ubiquitous computing and various sensor technologies, we are observing a massive population of multimodal sensors at the edge which posts significant challenges in fusing the data. In this poster we propose MultimodalHD, a novel Hyperdimensional Computing (HD)-based design for learning from multimodal data on edge devices. We use HD to encode raw sensory data to high-dimensional low-precision hypervectors, after which the multimodal hypervectors are fed to an attentive fusion module for learning richer representations via inter-modality attention. Our experiments on multimodal time-series datasets show MultimodalHD to be highly efficient. MultimodalHD achieves 17x and 14x speedup in training time per epoch on HAR and MHEALTH datasets when comparing with state-of-the-art RNNs, while maintaining comparable accuracy performance.

Recent years have witnessed a significant expansion in Internet-of-Things (IoT) applications. Although the battery energy availability can be improved with energy harvesting, the overall device reliability management has been overlooked in the existing literature. State-of-the-art reliability models of solar panels, electronics and rechargeable batteries show exponential dependence of failures on temperature. This work is the first to develop a comprehensive reliability deployment framework for energy-harvesting IoT networks, reflecting the non-negligible thermal stresses on each hardware component. Our framework improves the reliability on both pre-deployment and post-deployment stages. Prior to deployment, given the historical temperature and solar radiation of the region, we formulate a Mixed Integer Linear Program (MILP) to place the minimum number of nodes, while ensuring (i) full target coverage, (ii) complete connectivity, (iii) energy-neutral operation, and (iv) reliability constraints at each deployed node. We propose a polynomial-time heuristic, R-TSH, to approximate the optimal placement in large-scale deployments. While R-TSH optimizes long-term reliability, the prompt temperature or link quality differences from the historical patterns can significantly degrade device reliability after deployment. The post-deployment section of our design consists of a reliability-driven routing algorithm, AODV-Rel, that adapts to real-time environmental and link quality changes. Extensive analysis is done using a real-world dataset from the National Solar Radiation Database. Simulations in ns-3 show that R-TSH meets all reliability constraints even after 5 years of deployment as compared to the state of the art. In addition, it is 2000x faster than the optimal solution, while placing only 28% more nodes. AODV-Rel further extends the minimal operational lifetime by 1.5 and 2.8 months under temperature deviation and wireless interference.

Federated Learning (FL) is a widely adopted distributed learning paradigm for to its privacy-preserving and collaborative nature. In FL, each client trains and sends a local model to the central cloud for aggregation. However, FL systems using neural network (NN) models are expensive to deploy on constrained edge devices regarding computation and communication. In this demo, we present FedHD, a FL system using Hyperdimensional Computing (HDC). In contrast to NN, HDC is a brain-inspired and lightweight computing paradigm using high-dimensional vectors and associative memory. Our measurements indicate that FedHD is 3.2×, 3.2×, 5× better on performance, energy and communication efficiency respectively compared to NN-based FL systems whilst maintaining similar accuracy to the state of the art. Our code is available on GitHub1.

Low-Power Wide-Area Networks (LPWAN) has recently been scaling rapidly, targeting at large-scale and low-power applications. LoRa and LoRaWAN have been adopted in many practical deployments. While the advantages of LoRa have been well-demonstrated, challenges in scalability and reliability impede LoRa networks from further expansion. Traditional deployment strategies for reliability and fault tolerance, which ensure that multiple networking paths are available, are not directly applicable because of LoRa's single-hop and Aloha medium-access design. In this paper, we study how to design LoRa networks in large regions so that their transmission reliability and fault tolerance against gateway failures and interference are met for LoRaWAN technology. We first introduce m-gateway connectivity to guarantee fault tolerance due to LoRa's unique properties. Next, we leverage state-of-the-art transmission reliability model based on estimated path loss from satellite maps. Combining the above two contributions, we formulate an Integer Nonlinear Program (INLP) that minimizes the number of gateways through strategic gateway placement and resource allocation. Constraints are imposed to achieve (i) fault tolerance, (ii) reliable transmission (i.e., satisfactory QoS), (iii) sufficiently long lifetime. Due to high complexity of INLP, we design a greedy heuristic, RFT-LoRa, to acquire a high-quality solution for larger size problems. Comprehensive evaluation is performed with ns-3 simulator using real-world datasets. The results demonstrate that RFT-LoRa enhances average packet delivery ratio by 10% - 54% over the existing heuristic under gateway failures and interference.

Recent years have witnessed a significant expansion in Internet-of-Things (IoT) applications, especially in environmental monitoring, which aims at providing full coverage over potential targets. With energy harvesting ability, sensor devices can be replenished by external energy sources, and thus their lifetime is prolonged. While existing literature focuses on minimizing deployment costs, the reliability management is overlooked. Previous research has addressed that a higher temperature exponentially accelerates hardware failure rates. The versatile outdoor environments impose a non-negligible thermal stress on the hardware and consequently reduce the reliability of devices. In this paper, we are the first to propose a reliability-driven sensor deployment approach to achieve minimum nodes, while satisfying (i) full target coverage, (ii) complete connectivity, (iii) energy-neutral operation, and (iv) reliability constraints. Given external temperature distribution, we propose an algorithm to convert reliability constraints to a single-value power threshold for each location. A Mixed Integer Linear Programming (MILP) model is formulated and solved with CPLEX. Due to the complex nature of MILP, we propose a heuristic, named Reliability-driven TwoStage Heuristic (R-TSH), to approximate the optimal solution for large-scale problems. Extensive simulations are performed on a real-world dataset from the National Solar Radiation Database. Our results indicate that R-TSH meets all reliability constraints with only 20% more sensors than the optimal solution, while executing more than 1500x faster. Compared to state-of-the-art heuristics, R-TSH avoids 20 - 80% of reliability violations with a comparable number of nodes and execution time.

Mobile edge computing is an emerging research topic which aims at pushing the computation from the cloud to the edge devices. Most of the current machine learning (ML) algorithms, such as federated learning, are designed for homogeneous mobile networks, that is, all the devices collect the same type of data. In this paper, we address distributed training of ML algorithms in heterogeneous mobile networks where the features, rather than the samples, are distributed across multiple heterogeneous mobile devices. Training ML models in heterogeneous mobile networks incurs a large communication cost due to the necessity to deliver the local data to a central server. Inspired by active learning, which is traditionally used to reduce the labeling cost for training ML models, we propose an active sampling method to reduce the communication cost of learning in heterogeneous mobile networks. Instead of sending all the local data, the proposed active sampling method identifies and sends only informative data from each device to the central server. Extensive experiments on four real datasets, both with numerical simulation and on a networked mobile system, show that the proposed method can reduce the communication cost by up to 53% and energy consumption by up to 67% without accuracy degradation compared with the conventional approaches.

Recent advances in low-power long-range communication schemes such as LoRa have opened up new potentials in large-scale Internet-of-Things (IoT) applications, especially environmental monitoring. However, the versatile environment and the long traveling distance have imposed significant challenges to maintenance. Previous research has shown that higher temperature exponentially accelerates electronics failure rates. The maintenance cost can take as much as 80% of the total deployment expenses if not managed carefully. In this article, we formulate a sensor deployment problem to preventively minimize maintenance costs while ensuring tolerable sensing quality and complete connectivity. We are the first to derive a maintenance cost model for IoT networks considering thermal degradation and battery depletion. To assess the spatial phenomena of interest, we adopt the sensing quality metric based on mutual information. While the proposed problem is nonconvex, we bring up a relaxed form and solve it with a sparse nonlinear optimizer. We further apply two population-based metaheuristics, i.e., particle swarm optimization (PSO) and artificial bee colony (ABC) algorithm, to approximate the optimal solution. Extensive simulations are performed on two real-world datasets of the Southern California region in the U.S. Our metaheuristics save up to 40% of maintenance cost compared with the existing greedy heuristics under the same acceptable sensing quality.

Considering its advantages in dealing with high-dimensional visual input and learning control policies in discrete domain, Deep Q Network (DQN) could be an alternative method of traditional auto-focus means in the future. In this paper, based on Deep Reinforcement Learning, we propose an end-to-end approach that can learn auto-focus policies from visual input and finish at a clear spot automatically. We demonstrate that our method - discretizing the action space with coarse to fine steps and applying DQN is not only a solution to auto-focus but also a general approach towards vision-based control problems. Separate phases of training in virtual and real environments are applied to obtain an effective model. Virtual experiments, which are carried out after the virtual training phase, indicates that our method could achieve 100% accuracy on a certain view with different focus range. Further training on real robots could eliminate the deviation between the simulator and real scenario, leading to reliable performances in real applications.