Abstract Foundation models have gained growing interest in the IoT domain due to their reduced reliance on labeled data and strong generalizability across tasks, which address key limitations of traditional machine learning approaches. However, most existing foundation model based methods are developed for specific IoT tasks, making it difficult to compare approaches across IoT domains and limiting guidance for applying them to new tasks. This survey aims to bridge this gap by providing a comprehensive overview of current methodologies and organizing them around four shared performance objectives by different domains: efficiency , context-awareness , safety , and security & privacy . For each objective, we review representative works, summarize commonly-used techniques and evaluation metrics. This objective-centric organization enables meaningful cross-domain comparisons and offers practical insights for selecting and designing foundation model based solutions for new IoT tasks. We conclude with key directions for future research to guide both practitioners and researchers in advancing the use of foundation models in IoT applications.

In this work, we develop an end-to-end differentiable multi-view visual tracking architecture and explore fine-tuning model parameters via gradient-based optimization and automatic differentiation. We consider a setting with multiple camera nodes distributed in the tracking environment that collaboratively track objects. The architecture that we construct includes within-image-plane deep learning-based detection models, probabilistic camera models, object dynamics models, and an$N$-object Kalman filter-based tracking model. We demonstrate fully differentiable choices for each of these components, enabling learning and fine-tuning of the parameters of all system components based on different forms of supervision. Our results show performance gains for$N$-object tracking when fine-tuning the parameters of the system for end-to-end tracking performance.

The widespread adoption of Internet of Things (IoT) devices has significantly enhanced convenience for consumers, yet the privacy implications of these devices remain unclear to most users, even with the availability of privacy policies. To address this challenge, we introduce a novel visualization tool that provides an informative and expressive visual representation of the sensors, data processing workflows, and associated privacy risks of IoT devices. This user-friendly tool is designed to enhance user understanding, empowering them to make informed decisions about their privacy.

While existing strategies to execute deep learning-based classification on low-power platforms assume the models are trained on all classes of interest, this paper posits that adopting context-awareness i.e. narrowing down a classification task to the current deployment context consisting of only recent inference queries can substantially enhance performance in resource-constrained environments. We propose a new paradigm, CACTUS, for scalable and efficient context-aware classification where a micro-classifier recognizes a small set of classes relevant to the current context and, when context change happens (e.g., a new class comes into the scene), rapidly switches to another suitable micro-classifier. CACTUS features several innovations, including optimizing the training cost of context-aware classifiers, enabling on-the-fly context-aware switching between classifiers, and balancing context switching costs and performance gains via simple yet effective switching policies. We show that CACTUS achieves significant benefits in accuracy, latency, and compute budget across a range of datasets and IoT platforms.

Internet of Things (IoT) provides a wide range of services to improve convenience and comfort in our daily lives. However, various sensors equipped on IoT devices often raise privacy concerns. Prior works on privacy focus on passive protection from the data and device perspective, such as data encryption and communication protocol design. In this work, we introduce PrivaSee, an augmented reality (AR)-enabled privacy visualization platform to empower users with proactive privacy protection by enhancing their understanding of privacy perception for multimodal sensors.

Visual object tracking has seen significant progress in recent years. However, the vast majority of this work focuses on tracking objects within the image plane of a single camera and ignores the uncertainty associated with predicted object locations. In this work, we focus on the geospatial object tracking problem using data from a distributed camera network. The goal is to predict an object’s track in geospatial coordinates along with uncertainty over the object’s location while respecting communication constraints that prohibit centralizing raw image data. We present a novel single-object geospatial tracking data set that includes high-accuracy ground truth object locations and video data from a network of four cameras. We present a modeling framework for addressing this task including a novel backbone model and explore how uncertainty calibration and fine-tuning through a differentiable tracker affect performance.

Sensing and data collection platforms are the crucial components of high-quality datasets that can fuel advancements in research. However, such platforms usually are ad-hoc designs and are limited in sensor modalities. In this paper, we discuss our experience designing and deploying a multi-modal multi-node sensor data collection platform that can be utilized for various data collection tasks. The main goal of this platform is to create a modality-rich data collection platform suitable for Internet of Things (IoT) applications with easy reproducibility and deployment, which can accelerate data collection and downstream research tasks.

Human sensing, motion trajectory estimation, and identification are central to a wide range of applications in many domains such as retail stores, surveillance, public safety, public address, smart homes and cities, and access control. Existing solutions either require facial recognition or installation and maintenance of multiple units, or they lack long-term re-identification capability. In this paper, we propose a novel system - called EyeFi- that combines WiFi and camera on a standalone device to overcome these limitations. EyeFi integrates a WiFi chipset to an overhead camera and fuses motion trajectories obtained from both vision and RF modalities to identify individuals. In order to do that, EyeFi uses a student-teacher model to train a neural network to estimate the Angle of Arrival (AoA) of WiFi packets from the CSI values. Based on extensive evaluation using real-world data, we observe that EyeFi improves WiFi CSI based AoA estimation accuracy by more than 30% and offers 3,800 times computational speed over the state-of-the-art solution. In a real-world environment, EyeFi's accuracy of person identification averages 75% when the number of people varies from 2 to 10.

Human sensing, motion tracking, and identification are at the center of numerous applications such as customer analysis, public safety, smart cities, and surveillance. To enable such capabilities, existing solutions mostly rely on vision-based approaches, e.g., facial recognition that is perceived to be too privacy invasive. Other camera-based approaches using body appearances lack long-term re-identification capability. WiFi-based approaches require the installation and maintenance of multiple units. We propose a novel system - called EyeFi [2] - that overcomes these limitations on a standalone device by fusing camera and WiFi data. We use a three-antenna WiFi chipset to measure WiFi Channel State Information (CSI) to estimate the Angle of Arrival (AoA) using a neural network trained with a novel student-teacher model. Then, we perform cross modal (WiFi, camera) trajectory matching to identify individuals using the MAC address of the incoming WiFi packets. We demonstrate our work using real-world data and showcase improvements over traditional optimization-based methods in terms of accuracy and speed.

In this paper, we present - ZenCam, which is an always-on body camera that exploits readily available information in the encoded video stream from the on-chip firmware to classify the dynamics of the scene. This scene-context is further combined with simple inertial measurement unit (IMU)-based activity level-context of the wearer to optimally control the camera configuration at run-time to keep the device under the desired energy budget. We describe the design and implementation of ZenCam and thoroughly evaluate its performance in real-world scenarios. Our evaluation shows a 29.8-35% reduction in energy consumption and 48.1-49.5% reduction in storage usage when compared to a standard baseline setting of 1920×1080 at 30fps while maintaining a competitive or better video quality at the minimal computational overhead.

This paper introduces a system that takes radio frequency (RF) signals from an off-the-shelf, low-cost, 77 GHz mm Wave radar and produces an enhanced 3D RF representation of a scene. Such a system can be used in scenarios where camera and other types of sensors do not work, or their performance is impacted due to bad lighting conditions and occlusions, or an alternate RF sensing system like synthetic aperture radar (SAR) is too large, inconvenient, and costly. The enhanced RF representation can be used in numerous applications such as robot navigation, human-computer interaction, and patient monitoring. We use off-the-shelf parts to capture RF signals and collect our own data set for training and testing of the approach. The novelty of the system lies in its use of AI to generate a fine-grained 3D representation of an RF scene from its sparse RF representation which a mm Wave radar of the same class cannot achieve.

Low swing TSV signaling is proposed for three-dimensional (3D) integrated circuits (ICs) to reduce dynamic power consumption. Novel level shifters are designed to lower the voltage swing before the TSV and to pull the voltage swing back to full rail at the far end of the TSV. Proposed level shifters operate with a single supply voltage, thereby reducing the overall cost. Critical TSV capacitance beyond which the proposed scheme saves dynamic power is determined. Up to 42% reduction in overall power is demonstrated with a voltage swing of 0.5 V, where the supply voltage is 1 V.