Talking-face synthesis is the process of generating facial motions synchronized with given audio, serving as an enabling technology for next-generation virtual assistants and video conferencing. However, existing talking-face synthesis systems rely on computation-intensive generative models, posing significant challenges for resource-constrained devices such as smartphones and Personal Digital Assistants (PDAs). This paper presents FastAvatar, an efficient talking-face synthesis system tailored for resource-limited devices. FastAvatar builds on our observations of the unique structure of generative models and the correlation between audio inputs and motion outputs. Specifically, FastAvatar employs a multimodal caching mechanism to avoid redundant computations and a fast adaptation strategy to handle fluctuating workloads, significantly improving end-to-end responsiveness. Our implementation and evaluation on multiple devices demonstrate that FastAvatar reduces latency by 3 times, with slight quality degradation (7.9 % LSE-C and 6.8 % PSNR). To the best of our knowledge, FastAvatar is the first work to support talking-face synthesis on resource-constrained devices.

Supply chain management (SCM), a critical factor in enhancing companies’ efficiency and competitiveness, has received significant attention from both industry and academia. Beyond the flow of products from supplier to customer, SCM also involves the flow of information necessary to monitor, track, and optimize the entire product lifecycle. Consequently, integrating the information flow in SCM with the Internet of Things (IoT) is imperative, as IoT provides the ability to onboard, interconnect, interact with, and sense products. However, IoT-enabled SCM also presents unique challenges, such as managing large volumes of data, ensuring credible and traceable data sources, and achieving low-cost, seamless connectivity. In this article, we systematically present recent advances in leveraging IoT to build robust and effective SCM systems. Unlike existing surveys and overviews, this article focuses on the customer side as customers are the destination of the information flow and tightly coupled with IoT networks. We offer deep insights into the principles, challenges, and research opportunities in IoT-enabled SCM, aiming to assist IoT practitioners in understanding and designing IoT solutions for SCM.

Recently, there has been an increasing drive to consolidate multiple microcontrollers into one physical entity, due to advantages in reducing overall costs, enhancing reliability, and simplifying hardware interconnections. To reduce consolidation engineering costs, minimizing system latency and memory footprint is important as well as maintaining compatibility with legacy software. In this paper, we propose a virtualization-based solution called Feather-weight Virtual Machine (FVM) that focuses on these goals.FVMenables low latency by specializing the virtualization model to Real-Time Operating Systems (RTOSes), achieves small footprint by adapting management policies to microcontroller memories, attains high compatibility by aligning with microcontroller ecosystem idiosyncrasies, finally allowing practical consolidation across a wide range of commodity microcontrollers. We implement and evaluateFVMon ARMv6-M, ARMv7-M, and RISC-V architectures with two toolchains and two RTOSes, and it can fit into 20 KiB of RAM with less than 5% latency bloat.

Neural network models have become integral to Internet of Things (IoT) systems, with applications spanning from industrial automation to critical infrastructure management. Despite their prevalence, the deployment of these models within IoT systems introduces distinctive security vulnerabilities. In particular, adversaries may execute model poisoning attacks, which aim to alter the decision-making processes of embedded models, leading to erroneous outcomes. Existing model poisoning attacks necessitate access to extensive auxiliary datasets, such as the training dataset itself or one with same distribution. These requirements often render such attacks impractical in IoT contexts, given the constrained storage and computational resources of IoT devices. This paper proposes the first model poisoning attack against interpreters without auxiliary datasets to manipulate the model’s behavior. We evaluate the attack on three real-world datasets, and results indicate that this attack can successfully coerce the targeted interpreters to produce outcomes aligned with an adversary’s intentions, while maintaining nearly indistinguishable performance from the original model, thereby ensuring its stealthiness. Furthermore, beyond directly affected interpreters, our experiments reveal that four additional interpreters coupled to the poisoned model are indirectly influenced, underscoring the attack’s transferability.

Internet of Things (IoT) devices are frequently deployed in highly dynamic environments and need to continuously learn new classes from data streams. Incremental Learning (IL) has gained popularity in IoT as it enables devices to learn new classes efficiently without retraining model entirely. IL involves fine-tuning the model using two sources of data: a small amount of representative samples from the original training dataset and samples from the new classes. However, both data sources are vulnerable to Membership Inference Attack (MIA). Fortunately, the existing MIAs result in poor performance against IL, because they ignore features such as the similarity between old and new models at the old classification layer. This paper presents the first MIA against IL, capable of determining not only whether a sample was used for training/fine-tuning but also distinguishing whether it belongs to the representative dataset or the new classes (unique in IL). Extensive experiments validate the effectiveness of our attack across four real-world datasets. Our attack achieves an average attack success rate of 74.03% in the white-box setting (model structure and parameters are known) and 70.08% in the black-box setting. Importantly, our attack is not sensitive to the IL hyper-parameters (e.g., distillation temperature), confirming its accurate, robust, and practical.

The widespread adoption of WiFi has driven numerous WiFi-based wireless sensing applications. Researchers have utilized Channel State Information (CSI) from WiFi communications to develop various applications and systems, such as activity recognition, gesture recognition, and user authentication. However, unlike the payload data of packets, the CSI in WiFi packet headers lacks effective encryption mechanisms, posing a risk of privacy leakage. This paper proposes a privacy protection method for WiFi-based wireless sensing applications by controlling CSI through the modification of pilot symbols, specifically the Long Training Sequence (LTS), in WiFi packet headers. This approach affects the results of wireless sensing applications. To achieve encrypted protection of CSI, we designed a virtual channel model implemented at the FPGA level based on the OpenWiFi architecture. We simulate multipath effects to encrypt the IQ signal before its analog conversion. After passing through this virtual channel, the signal is transmitted through the real physical channel and collected by the receiver. Additionally, to further protect data privacy, we implemented a targeted protection algorithm for customized precise control of CSI in the current physical environment. Based on the current CSI data and the target CSI data, the algorithm customizes the parameter selection of the virtual channel model to generate a specific virtual channel for targeted encryption. Finally, our research verified the existence of privacy leakage issues in wireless sensing systems through experiments on respiration detection and human activity recognition based on CSI. We also validated the effectiveness of our designed virtual channel model in privacy protection.

WiFi sensing is critical to many applications, such as localization, human activity recognition, and contact-less health monitoring. With metaverse and ubiquitous sensing advances, WiFi sensing becomes increasingly imperative. However, as shown in this paper, WiFi sensing data leaks users’ private attributes (e.g., height, weight, and gender), violating increasingly stricter privacy protection laws and regulations. To demonstrate the leakage of private attributes in WiFi sensing, we investigate two public WiFi sensing datasets and apply a deep learning model to recognize users’ private attributes. Our experimental results clearly show that our model can identify users’ private attributes in WiFi sensing data collected by general WiFi applications, with almost 100% accuracy for gender inference, less than 4 cm error for height inference, and about 4 kg error for weight inference, respectively. Our finding calls for research efforts to preserve data privacy while enabling WiFi sensing-based applications.

As acoustic communication systems become increasingly common in our daily life, eavesdropping brings severe security and privacy risks. Current methods of acoustic eavesdropping either provide low resolution due to the use of sub-6 GHz frequencies, work only for limited words based on classification approaches, or cannot work through-wall because of the use of optical sensors. In this article, we presentmilliEar, a mmWave acoustic eavesdropping system that leverages the high-resolution of mmWave FMCW ranging and generative machine learning models to not only extract vibrations but to reconstruct the audio.milliEarcombines speaker vibration estimation with conditional generative adversarial networks to eavesdrop and recover high-quality audios (i.e., with no vocabulary constraints). We implement and evaluatemilliEarusing off-the-shelf mmWave radars deployed in different scenarios and settings. Evaluation results clearly show thatmilliEarcan accurately reconstruct the audio even at different distances, angles, and through the wall with different insulator materials. In addition, our subjective and objective evaluations demonstrate that the reconstructed audio has a strong similarity with the original audio.

We propose MASTAF, a Model-Agnostic Spatio-Temporal Attention Fusion network for few-shot video classification. MASTAF takes input from a general video spatial and temporal representation,e.g., using 2D CNN, 3D CNN, and Video Transformer. Then, to make the most of such representations, we use self- and cross-attention models to highlight the critical spatio-temporal region to increase the inter-class variations and decrease the intra-class variations. Last, MASTAF applies a lightweight fusion network and a nearest neighbor classifier to classify each query video. We demonstrate that MASTAF improves the state-of-the-art performance on three few-shot video classification benchmarks(UCF101, HMDB51, and Something-Something-V2), e.g., by up to 91.6%, 69.5%, and 60.7% for five-way one-shot video classification, respectively.

Deep Learning (DL) has been widely adopted in almost all domains, from threat recognition to medical diagnosis. Albeit its supreme model accuracy, DL imposes a heavy burden on devices as it incurs overwhelming system overhead to execute DL models, especially on Internet-of-Things (IoT) and edge devices. Collaborative inference is a promising approach to supporting DL models, by which the data owner (the victim) runs the first layers of the model on her local device and then a cloud provider (the adversary) runs the remaining layers of the model. Compared to offloading the entire model to the cloud, the collaborative inference approach is more data privacy-preserving as the owner’s model input is not exposed to outsiders. However, we show in this paper that the adversary can restore the victim’s model input by exploiting the output of the victim’s local model. Our attack is dubbed Ginver 1: Generative model inversion attacks against collaborative inference. Once trained, Ginver can infer the victim’s unseen model inputs without remaking the inversion attack model and thus has the generative capability. We extensively evaluate Ginver under different settings (e.g., white-box and black-box of the victim’s local model) and applications (e.g., CIFAR10 and FaceScrub datasets). The experimental results show that Ginver recovers high-quality images from the victims.

As a privacy-preserving paradigm for training machine learning (ML) models, federated learning (FL) has received tremendous attention from both industry and academia. In a typical FL scenario, clients exhibit significant heterogeneity in terms of data distribution and hardware configurations. Thus, randomly sampling clients in each training round may not fully exploit the local updates from heterogeneous clients, resulting in lower model accuracy, slower convergence rate, degraded fairness, etc. To tackle the FL client heterogeneity problem, various client selection algorithms have been developed, showing promising performance improvement. In this article, we systematically present recent advances in the emerging field of FL client selection and its challenges and research opportunities. We hope to facilitate practitioners in choosing the most suitable client selection mechanisms for their applications, as well as inspire researchers and newcomers to better understand this exciting research topic.

Federated learning (FL) is a distributed model training paradigm that preserves clients’ data privacy. It has gained tremendous attention from both academia and industry. FL hyper-parameters (e.g., the number of selected clients and the number of training passes) significantly affect the training overhead in terms of computation time, transmission time, computation load, and transmission load. However, the current practice of manually selecting FL hyper-parameters imposes a heavy burden on FL practitioners because applications have different training preferences. In this paper, we propose, an automatic FL hyper-parameter tuning algorithm tailored to applications’ diverse system requirements in FL training. iteratively adjusts FL hyper-parameters during FL training and can be easily integrated into existing FL systems. Through extensive evaluations of for diverse applications and FL aggregation algorithms, we show that is lightweight and effective, achieving 8.48%-26.75% system overhead reduction compared to using fixed FL hyper-parameters. This paper assists FL practitioners in designing high-performance FL training solutions. The source code of is available at.

Due to the substantial computational cost of neural network training, adopting third-party models has become increasingly popular. However, recent works demonstrate that third-party models can be poisoned. Nonetheless, most model poisoning attacks require reference data, e.g., training dataset or data belonging to the target label, making them difficult to launch in practice. In this paper, we propose a reference data independent model poisoning attack that can (1) directly search for sensitive features with respect to the target label, (2) quantify the positive and negative effects of the model parameters on sensitive features, and (3) accomplish the training of poisoned model by our parameter selective update strategy. The extensive evaluation on datasets with a few classes and numerous classes show that the attack is (I) effective: the trigger input can be labeled as a deliberate class by the poisoned model with high probability; (II) covert: the performance of the poisoned model is almost indistinguishable from the intact model on non-trigger inputs; and (III) straightforward: an adversary only needs a little background knowledge to launch the attack. Overall, the evaluation results show that our attack achieves 95%, 100%, 81%, 96%, and 96% success rates on Cifar10, Cifar100, ISIC2018, FaceScrub, and ImageNet datasets, respectively.

Having an efficient onboarding process is a pivotal step to utilize and provision the IoT devices for accessing the network infrastructure. However, the current process to onboard IoT devices is time-consuming and labor-intensive, which makes the process vulnerable to human errors and security risks. In order to have a streamlined onboarding process, we need a mechanism to reliably associate each digital identity with each physical device. We design an onboarding mechanism called MAIDE to fill this technical gap. MAIDE is an Augmented Reality (AR)-facilitated app that systematically selects multiple measurement locations, calculates measurement time for each location and guides the user through the measurement process. The app also uses an optimized voting-based algorithm to derive the device-to-ID mapping based on measurement data. This method does not require any modification to existing IoT devices or the infrastructure and can be applied to all major wireless protocols such as BLE, and WiFi. Our extensive experiments show that MAIDE achieves high device-to-ID mapping accuracy. For example, to distinguish two devices on a ceiling in a typical enterprise environment, MAIDE achieves ~95% accuracy by measuring 5 seconds of Received Signal Strength (RSS) data for each measurement location when the devices are 4 feet apart.

In the past few years, the computer vision community has developed numerous novel technologies of 3D vision (e.g., 3D object detection and classification and 3D scene segmentation). In this work, we explore the opportunities brought by these innovations for enabling real-time 3D vision on mobile devices. Mobile 3D vision finds various use cases for emerging applications such as autonomous driving, drone navigation, and augmented reality (AR). The key differences between 3D vision and 2D vision mainly stem from the input data format (i.e., point clouds or 3D meshes vs. 2D images). Hence, the key challenge of 3D vision is that it is could be more computation intensive and memory hungry than 2D vision, due to the additional dimension of input data. For example, our preliminary measurement study of several state-of-the-art machine learning models for 3D vision shows that none of them can execute faster than one frame per second on smartphones. Motivated by these challenges, we present in this position paper a research agenda on offering systems support for real-time mobile 3D vision, focusing on improving its computation efficiency and memory utilization.

Three-Dimensional (3D) semantic segmentation is an essential building block for interactive Augmented Reality (AR). However, existing Deep Neural Network (DNN) models for segmenting 3D objects are not only computation-intensive but also memory heavy, hindering their deployment on resource-constrained mobile devices. We present the design, implementation and evaluation of Slimmer, a generic and model-independent framework for accelerating 3D semantic segmentation and facilitating its real-time applications on mobile devices. In contrast to the current practice that directly feeds a point cloud to DNN models, Slimmer is motivated by our observation that these models remain high accuracy even if we remove a fraction of points from the input, which can significantly reduce the inference time and memory usage of these models. Our design of Slimmer faces two key challenges. First, the simplification method of point clouds should be lightweight. Otherwise, the reduced inference time may be canceled out by the incurred overhead of input-data simplification. Second, Slimmer still needs to accurately segment the removed points from the input to create a complete segmentation of the original input, again, using a lightweight method. Our extensive performance evaluation demonstrates that, by addressing these two challenges, Slimmer can dramatically reduce the resource utilization of a representative DNN model for 3D semantic segmentation. For example, if we can tolerate 1% accuracy loss, the reduction could be ~20% for inference time and ~9% for memory usage. The reduction increases to around ~27% for inference time and ~15% for memory usage when we can tolerate 2% accuracy loss.

As a massive number of the Internet of Things (IoT) devices are deployed, the security and privacy issues in IoT arouse more and more attention. The IoT attacks are causing tremendous loss to the IoT networks and even threatening human safety. Compared to traditional networks, IoT networks have unique characteristics, which make the attack detection more challenging. First, the heterogeneity of platforms, protocols, software, and hardware exposes various vulnerabilities. Second, in addition to the traditional high-rate attacks, the low-rate attacks are also extensively used by IoT attackers to obfuscate the legitimate and malicious traffic. These low-rate attacks are challenging to detect and can persist in the networks. Last, the attackers are evolving to be more intelligent and can dynamically change their attack strategies based on the environment feedback to avoid being detected, making it more challenging for the defender to discover a consistent pattern to identify the attack. In order to adapt to the new characteristics in IoT attacks, we propose a reinforcement learning-based attack detection model that can automatically learn and recognize the transformation of the attack pattern. Therefore, we can continuously detect IoT attacks with less human intervention. In this paper, we explore the crucial features of IoT traffics and utilize the entropy-based metrics to detect both the high-rate and low-rate IoT attacks. Afterward, we leverage the reinforcement learning technique to continuously adjust the attack detection threshold based on the detection feedback, which optimizes the detection and the false alarm rate. We conduct extensive experiments over a real IoT attack data set and demonstrate the effectiveness of our IoT attack detection framework.

LED-to-camera communication allows LEDs deployed for illumination purposes to modulate and transmit data which can be received by camera sensors available in mobile devices like smartphones, wearable smart-glasses, etc. Such communication has a unique property that a user can visually identify a transmitter (i.e., LED) and specifically receive information from the transmitter. It can support a variety of novel applications such as augmented reality through mobile devices, navigation using smart signs, fine-grained location specific advertisement, etc. However, the achievable data rate in current LED-to-camera communication techniques remains very low to support any practical application. In this paper, we present ColorBars, an LED-to-camera communication system that utilizes Color Shift Keying (CSK) to modulate data using different colors transmitted by the LED. It exploits the increasing popularity of Tri-LEDs (RGB) that can emit a wide range of colors. We show that commodity cameras can efficiently and accurately demodulate the color symbols. ColorBars ensures flicker-free and reliable communication even in the presence of inter-frame loss and diversity of rolling shutter cameras. We implement ColorBars on embedded platform and evaluate it with Android and iOS smartphones as receivers. Our evaluation shows that ColorBars can achieve a data rate of 7.7 Kbps on Nexus 5, 3.7 Kbps on iPhone 5S, and 2.9 Kbps on Samsung Note8. It is also shown that lower CSK modulations (e.g., four and eight CSK) provide extremely low symbol error rates (-3), making them a desirable choice for reliable LED-to-camera communication.

This paper presents StrLight, the first practical VLC system that leverages widely-deployed string lights to transmit data. The data transmission is imperceptible to human eyes. Users can decode the data by mobile devices (e.g., smartphones) equipped with cameras. StrLight primarily differs from existing VLC systems in using string lights which are composed of a large number of small LEDs and thus the unique design to address practical issues including a special data modulation/encoding scheme, a data representation with unstructured/unknown topologies of LEDs in the string light, and a fault tolerance against broken and blocked LEDs. To the best of our knowledge, StrLight is the first practical VLC system of its kind. We build several prototypes of string light transmitters and test with different smartphone models and a customized mobile device as receivers. The experiment results show that StrLight provides an efficient and robust data broadcasting. A string light of 100 LEDs working in 450 Hz and a camera with a capture rate of 30 Hz and an image resolution of as low as 320 × 240 pixels, delivers data rate of ~1 kbps, without observable light flickers.

In this paper, we investigate whether IEEE 802.11ac WiFi can support VR applications. To this end we conduct a controlled study of WiFi performance in an indoor setting. Our measurements reveal that WiFi transmissions suffer from high latency and jitter, which makes WiFi systems inadequate for VR applications. For example, the round-trip delay can be as high as 228ms, and more than 24.2% packets experience jitter higher than 1ms. To locate the root cause of the high latency and jitter, we dissect the network stack layer by layer and find that the main culprit is the wireless channel transmission time. To reduce the channel transmission time, we propose using multiple network interfaces running on non-overlapping channels. By using only two interfaces, we (1) reduce the median round-trip delay by 28.6% and jitters of higher than 1ms by 11.5% compared to the best single interface in UDP transmissions, and (2) reduce the median round-trip delay by 38.9% in TCP transmissions. We believe that this paper sheds some light on whether we can make today's WiFi systems VR-ready by using multiple interfaces.

This paper presents DopEnc, an acoustic-based encounter profiling system on commercial off-the-shelf smartphones. DopEnc automatically identifies the persons that users interact with in the context of encountering. DopEnc performs encounter profiling in two major steps: (1) Doppler profiling to detect that two persons approach and stop in front of each other via an effective trajectory, and (2) voice profiling to confirm that they are thereafter engaged in an interactive conversation. DopEnc is further extended to support parallel acoustic exploration of many users by incorporating a unique multiple access scheme within the limited inaudible acoustic frequency band. All implementation of DopEnc is based on commodity sensors like speakers, microphones, and accelerometers integrated on mainstream smartphones. We evaluate DopEnc with detailed experiments and a real use-case study of 11 participants. Overall DopEnc achieves an accuracy of 6.9 percent false positive and 9.7 percent false negative in real usage.

This paper presents Pando, a completely contention-free data dissemination protocol for wireless sensor networks. Pando encodes data by Fountain codes and disseminates the rateless stream of encoded packets along the fast and parallel pipelines built on constructive interference and channel diversity. Since every encoded packet contains innovative information to the original data object, Pando avoids duplicate retransmissions and fully exploits the wireless broadcast effect in data dissemination. To transform Pando into a practical system, we devise several techniques, including the integration of Fountain coding with the timing-critical operations of constructive interference and pipelining, a silence-based feedback scheme for the one-way pipelined dissemination, and packet-level adaptation of network density and channel diversity. Based on these techniques, Pando can accomplish data dissemination entirely over the fast and parallel pipelines. We implement Pando in Contiki and for TelosB motes. We evaluate Pando with various settings on two large-scale open test beds, Indriya and Flocklab. Our experimental results show that Pando can provide 100% reliability and reduce the dissemination time of state of the art by 3.5×.

This paper presents DopEnc, an acoustic-based encounter profiling system on smartphones. DopEnc can automatically identify the persons that users interact with in the context of encountering. DopEnc performs encounter profiling in two major steps: (1) Doppler profiling to detect that two persons approach and stop in front of each other via an effective trajectory, and (2) voice profiling to confirm that they are thereafter engaged in an interactive conversation. DopEnc is further extended to support parallel acoustic exploration of many users by incorporating a unique multiple access scheme within the limited inaudible acoustic frequency band. All implementation of DopEnc is based on commodity sensors like speakers, microphones and accelerometers integrated on commercial-off-the-shelf smartphones. We evaluate DopEnc with detailed experiments and a real use-case study of 11 participants. Overall DopEnc achieves an accuracy of 6.9% false positive and 9.7% false negative in real usage.

This paper presents Pando, a completely contention-free data dissemination protocol for wireless sensor networks. Pando encodes data by Fountain codes and disseminates the rateless stream of encoded packets along the fast and parallel pipelines built on constructive interference and channel diversity. Since every encoded packet contains innovative information to the original data object, Pando avoids duplicate retransmissions and fully exploits the wireless broadcast effect in data dissemination. To transform Pando into a practical system, we devise several techniques, including the integration of Fountain coding with the timing-critical operations of constructive interference and pipelining, a silence based feedback scheme for the one-way pipelined dissemination, and packet-level adaptation of network density and channel diversity. Based on these techniques, Pando can accomplish the data dissemination process entirely over the fast and parallel pipelines. We implement Pando in Contiki and for TelosB sensor motes. We evaluate Pando's performance with various settings on two large-scale open testbeds, Indriya and Flocklab. Our experimental results show that Pando can provide 100% reliability and reduce the dissemination time of the state-of-the-art by 3.5.