With the rise of time-of-use and tiered electricity pricing, energy consumers are encouraged to adopt peak-shifting strategies by automatically controlling high-power appliances. These help lower energy costs while enhancing the power grid's stability. To support such energy management with high resilience and responsiveness, reliable short-term load forecasting (STLF) plays a critical role. STLF predicts electricity consumption over time horizons ranging from minutes to days, using historical data, temporal patterns, and contextual factors. Traditional top-down forecasting methods struggle to capture the complex consumption patterns of diverse and mixed appliance loads. Although bottom-up methods improve forecasting accuracy by integrating appliance-level data, monitoring all appliances is costly, and many do not meaningfully impact total load prediction. Therefore, we propose GCA-BULF, a bottom-up short-term load forecasting framework based on grouped critical appliances, supported by three key designs. First, the Critical Appliance Filtering module ranks appliances according to their power consumption, switching frequency, and usage pattern periodicity, and identifies critical ones through iterative load decomposition. Next, the Related Appliance Grouping module clusters these appliances based on spatial and temporal correlations for group-level forecasting. Finally, the Collaborative Load Forecasting module refines the total load prediction by combining multiple group-level forecasts. We evaluate GCA-BULF on residential and office building load forecasting tasks. Experimental results reveal that GCA-BULF improves hourly total load forecasting by 20.85%-57.88% compared to existing top-down methods and by 33.03%-92.48% compared to bottom-up methods.

Video has become increasingly widespread in information transmission and daily communication. However, video data often contain sensitive privacy information, which leads to privacy concerns. To address these concerns, video encryption has become one of the most effective privacy protection methods. Current efforts in video encryption focus on block-level or frame-level encryption, failing to provide fine-grained privacy protection. In response, we propose a personalized pixel-level video encryption approach for privacy protection. Our approach operates before encoding and is robust to lossy compression. It applies a quadruple encryption algorithm directly to the pixels within privacy-sensitive areas segmented by instance segmentation, achieving fine-grained, pixel-level encryption. Different encryption keys are managed using Attribute-Based Encryption (ABE) technology, which enables personalized access control. This approach assigns different permissions based on user identity, enabling multi-level access to encrypted video content.We deploy our approach across different devices and perform extensive experimental evaluations. Experimental results show that on different devices, the single-frame encryption overhead of our approach for videos of various resolutions consistently remains below 0.12 seconds, with the encrypted area achieving an average PSNR of 8.50 dB and an average SSIM of 0.079. When using a key with one bit difference from the correct key for decryption, the average number of pixels changing rate (NPCR) of the test sequence is 99.59% and the average unified average change intensity (UACI) is 32.40%. This shows that unauthorized users cannot obtain valid information when decrypting with an incorrect key.

End-edge collaborative inference improves computational efficiency by dividing a deep neural network into two parts, executed across the end device and the edge node in parallel. However, adversaries like malicious edge nodes can exploit transmitted data to reconstruct original inputs or infer sensitive attributes. Existing collaborative inference strategies upload the majority of input features to the edge node, significantly increasing the risk of privacy leakage, even without input reconstruction. Therefore, we propose PrivGuardInfer, a channel-level DNN end-edge collaborative inference strategy that optimizes intra-layer partition to simultaneously protect original inputs and sensitive attributes while ensuring latency constraints, supported by three key designs. First, the privacy measurements oriented both layer depth and channel count, jointly quantify the difficulty of reconstructing original inputs using varying numbers of feature maps across different layers. After assessing each channel's contribution, the information offset further measures the difficulty of inferring sensitive attributes. Finally, PrivGuardInfer models the privacy-optimal intra-layer partition under latency constraints as a grouped knapsack problem, mapping attack difficulty to item values and inference latency to item weights. Experimental results reveal that PrivGuardInfer achieves an average improvement of 80.54% in defending against model inversion attacks and 63.34% against attribute inference attacks compared to existing end-edge partition strategies. Moreover, it outperforms current privacy protection methods by an average of 69.37% and 49.75% in mitigating these two types of attacks.

The drive for privacy-preserving and low-latency artificial intelligence necessitates executing Large Language Models (LLMs) directly on heterogeneous, resource-constrained edge devices. This paradigm presents a challenge: efficiently running large models across multiple end/edge devices under a strict energy budget. Especially, the problem becomes more complicated when it comes to orchestrating multiple complex tasks using large models at the same time. To address this, we introduce FlexiTensor, an offline planning and deployment system for LLM inference, scaling from a single task to multiple concurrent tasks. FlexiTensor first considers LLM inference for a single task under a strict energy budget. We design a heuristic optimization algorithm to minimize the latency under a strict energy budget. Specifically, we select an optimized subset of devices and tensor allocations. Based on the selection, latency can be reduced by using non-uniform tensor parallelism with quantization. FlexiTensor can be extended to multi-task cases. We model the execution structure of multi-agent workflows as pre-defined Directed Acyclic Graphs (DAGs) and reformulate the problem with energy consumption constraints. FlexiTensor introduces a novel hybrid evolutionary algorithm to address joint task scheduling and resource allocation in multi-task cases. This approach navigates the vast search space of task placement, tensor splitting, and thread assignment to minimize the overall workflow makespan. All optimization algorithms run offline before deployment, using pre-profiled device characteristics to compute a static execution plan. Extensive experiments on a physical testbed of heterogeneous edge devices demonstrate that FlexiTensor significantly outperforms baselines. We accelerate single-task inference by up to 50%. For multi-task workflows, we consistently find superior scheduling solutions, achieving speed improvements by an average of 50% against competitive baselines and by up to 100% in certain scenarios, showcasing its effectiveness and adaptability for real-world edge LLM-based applications.

Programming-Community Question Answering (PCQA) aims to tackle issues through generating functional code and guiding descriptions. It involves multiple candidates, with different users having varying preferences for them. Additionally, one may contain outdated APIs. These undoubtedly present a challenge for responsing that meet user preferences. Recently, Reinforcement Learning from Human Feedback demonstrates its ability to precisely control the behavior of large language models (LLMs) to yield human-like responses. However, applying it to LLMs in domain-specific PCQA remains unexplored. In this work, we propose Multi-perspective Preference Alignment for Programming-Community Question Answering to generate user-centric responses, called MupPCQA. It includes three stages: Preference Standardization to control content quality, Preference Integration to consider diverse user tendencies, Preference Timeliness Mitigation to alleviate outdated answers. Extensive experiments on a high-quality, real-world PCQA dataset validate its accuracy and preference. Compared to its base model, MupPCQA shows an improvement of nearly 11% in BLEU, with increases of 20% and 17.5% in BERTScore and CodeBERTScore.

The high penetration of renewable energy presents significant challenges for the power grid in balancing supply and demand. Demand response (DR) is crucial for stable grid operation, yet existing research often overlooks actual electricity consumption patterns for the following day and the diversity among user types, compromising assessment accuracy and applicability. To address these shortcomings, this paper proposes an integrated load regulation system comprising “Identification-Prediction-Scheduling”. First, we identify specific user groups and utilize load prediction models to accurately forecast future loads. By analyzing predicted loads alongside historical data, we uncover electricity consumption patterns and DR potential for the following day, providing better insights for power grid scheduling. Finally, we implement a load control strategy aimed at optimizing grid stability. Our results show that the load forecasting model achieves average errors of 17.86% and 16.16% at intervals of 15 minutes and 1 hour, respectively. When demand is set at 1500 kW, the proposed load control strategy predicts a response load curtailment of 2216.85 kW and a DPI score of 50.75, significantly outperforming both the random selection strategy (1887.11 kW, DPI score of 18.39) and the priority to large FBC strategy (288.86 kW, DPI score of 40.78). This approach ensures better enterprise selection, contractual compliance, and improved benefits for electrical users.

To comply with the “right to be forgotten,” recent research has introduced machine unlearning techniques that enable machine learning models to remove specific data samples. However, existing multimodal machine unlearning has concerns about efficiency, and the model fidelity may deteriorate after unlearning, leading to meaningless outputs if given data samples that are requested to be forgotten. Instead of focusing on datalevel machine unlearning, we focus on information-level unlearning, aiming to forget specific information of data samples (such as sensitive information involving name or medical condition) while maintaining the model fidelity. In this work, we design an efficient multimodal machine unlearning (EMMU) framework to address model fidelity in vision-language systems. The core idea is to locate and only modify model parameters that are highly correlated with the specific information (which requires forgetting). EMMU locates crucial parameters associated with the sensitive information and updates these parameters using a multiobjective optimization strategy. Evaluations on vision-language tasks, such as Visual Question Answering and Image Captioning, using multiple datasets, demonstrate the efficiency and fidelity of EMMU. Compared to existing methods, our method obtains an average of$111 \times$improvement, boosting efficiency up to$1445 \times$. Meanwhile, the average utility-forget balance score has improved$9 \times$on average and reached up to$70 \times$, across multiple models and datasets.

The advent of a large language model (LLM) has revolutionized various domains and services. The inference pipeline system is emerging as an efficient mechanism to deploy LLMs. However, existing works barely study the deployment of LLM inference on heterogeneous GPUs (with different computation and memory capabilities), where inference efficiency can be heavily affected by the imbalanced performance of different pipeline stages. Based on our empirical experience, the unbalanced pipeline stages incur GPU wait time, and average idle time can exceed 50% of the whole LLM inference process. In this paper, we study and optimize the distributed pipeline parallelism system for LLM inference on heterogeneous GPUs. We present a heuristic algorithm and implement a system that automatically deploys an efficient inference pipeline on heterogeneous GPUs. Extensive experiments are evaluated on 26 heterogeneous GPUs. The results demonstrate the superiority of our proposed system, which improves makespan (i.e., the total LLM inference latency) and throughput by an average of 37.1% and a maximum of 83.0% compared to the baselines.

Graph prompt learning represents a novel paradigm aimed at enhancing the performance of graph learning models on a variety of downstream tasks by providing specific graph prompts. Despite its promise, current graph prompt learning methods are limited by the following limitations. On the one hand, existing methods often rely on manually selected graph information or simple learnable vectors, which can introduce human biases and lack expressiveness. These methods also fall short in guiding models to induce historical prior knowledge and improve generalization. Furthermore, the direct end-to-end tuning strategy of prompts lacks a necessary gentle transition, which impacts model stability and generalization. To overcome these limitations, we introduce the generalizable graph prompt learning framework (GGPL), which incorporates model-level prompt injection and a two-stage prompt tuning strategy. GGPL focuses on encoding subgraph structures and attributes during pre-training and uses SimGRACE to predict subgraph similarities, enhancing the base model's generalization. The model-level prompt injection module, with its prompt embedding backbone and self-prompt generation, seamlessly integrates invariant knowledge. Our two-stage tuning strategy, including transition and task-specific tuning, ensures better guidance and stability. By designing learnable prompt tokens and fine-tuning them with task-specific information, GGPL enables the model to generalize more robustly to downstream tasks. We conduct extensive experiments on six benchmark datasets to verify the model's effectiveness.

Object distance estimation is a critical task in autonomous driving and has received increasing attention in recent years. Existing methods rely on either geometric cues, which typically reduce structures to coarse attributes like 2D box dimensions, or visual cues that degrade under occlusion and truncation. In this work, we propose a geometry-enhanced framework that augments visual features with a more stable and descriptive geometric attribute—Patch-To-Center Geometry (PCG)—which encodes the spatial distance between local object patches and the 3D object center projection of the object to improve distance estimation. Specifically, each object is divided into multiple patches, and a dedicated attention module is employed to learn the pixel-wise spatial offsets from each patch to the projected center, serving as an auxiliary supervision signal. Additionally, a head token is introduced to aggregate global information from all patches for final distance prediction. Extensive experiments on the KITTI and nuScenes datasets demonstrate that our method outperforms existing approaches when limiting the minimum depth threshold.

The rapid growth of social media has led to the widespread uploading of private images to online networks, raising significant privacy concerns. Existing methods for image privacy assessment typically operate at coarse granularity, lack support for personalized settings, and primarily focus on explicit, entity-centered content. However, such approaches neglect implicit privacy, which refers to private information inferred from contextual cues rather than from any single identifiable visual entity, leading to a substantial underestimation of entire privacy risks. In this work, we propose SCOPE (Systematic Context-based Observation for Privacy Evaluation), a unified framework that systematically incorporates both explicit and implicit privacy across the entire image privacy lifecycle, including detection, quantitative risk assessment, and protection. SCOPE integrates context-aware image graphs with a concept-anchored ontology graph, enabling the incorporation of multi-source information to infer implicit privacy risks at both object and event levels. It further introduces novel qualitative and quantitative privacy metrics that jointly assess image-level privacy risks based on explicit and implicit content, and provides explainable mechanisms to guide implicit privacy protection. Experimental results demonstrate that SCOPE achieves 97.02% object-level and 88.37% event-level implicit privacy inference accuracy, outperforming previous methods by 15.88% and 21.83%, respectively. Extensive experiments and a user study further confirm the effectiveness of our privacy assessment metrics and protection mechanisms.

Secure protocol-independent communication is increasingly demanding to support information exchange among neighbor Internet of Things (IoT) devices. For example, recent works utilize ultrasound at the resonant frequency range of a gyroscope to build communication between a speaker and the gyroscope. However, they are vulnerable to eavesdropping attacks and may have limitations in communication delays. In this work, we present WordWhisper, an efficient, word-level, and speaker-to-gyroscope communication system, with which only the target device can receive the correct information. We theoretically analyze Micro-Electro-Mechanical System (MEMS) gyroscope resonance and propose a hardware-dependent mechanism to defend against eavesdropping, making non-target gyroscopes receive ineffective information. Note that WordWhisper is free of costly data collection from gyroscopes, we train and update our decoding model based on the synthesized data (generated from theoretical MEMS resonance analysis) rather than the costly collected data from gyroscopes. Meanwhile, we address the challenge of eavesdropping when it comes to multiple attackers. We evaluate WordWhisper over 50 MEMS gyroscopes and 100 words. Extensive evaluations demonstrate that WordWhisper can achieve word-level communication with 99.33% accuracy while the recognition accuracy drops to a random guess for the non-target. Our decoding delay is lower than 0.63 seconds.

Smart home technology has found wide-ranging applications in daily life, from enhancing energy efficiency to simplifying daily tasks and providing greater convenience. However, recent works have found that smart home devices are vulnerable to passive network observers (i.e., adversaries). While adversaries have demonstrated the ability to infer device events (e.g., whether a lamp is turned on) from the encrypted smart home traffic, we believe this only represents a less critical aspect of smart home privacy risks. Further analysis of demographic attributes presents greater risks to user privacy. Besides, from our deployment experience of real-world smart homes, we found that existing event inference methods can be greatly interfered with by event-unrelated traffic. Experiments show that this interference can result in up to a 10% drop in inference accuracy. Furthermore, it is challenging to infer finer-grained demographic attributes, due to the insufficient accuracy of event inference. Therefore, in this work, we propose a novel event inference model extracting multi-dimensional features that reduces the interference of event-unrelated traffic by analyzing packet length distribution and statistical properties. In addition, we design a dual-channel neural network to extract spatial and temporal relationships among triggered events to infer demographic attributes of smart home users, such as age group and career stage. Combining the above designs, we present TrafficDiary, the first user attribute inference approach based on smart home traffic traces. We prototype TrafficDiary and evaluate it in real-world smart homes. Experimental results show that TrafficDiary achieves 98.68% accuracy with a zero false positive rate in event inference and a high level of accuracy in user attribute inference, even when 16, 362 groups of event-unrelated traffic exist. TrafficDiary also performs well in terms of efficiency, with an inference latency of only 1.82 ms on a Raspberry Pi 4B device.

Deploying deep neural networks (DNNs) on IoT devices for model serving is a promising solution for intelligent applications with high real-time requirements and bandwidth sensitivity. To cope with the prohibitive computation and storage overheads of modern DNNs, great efforts have been devoted to the model compression technique. Most existing model compression approaches focus on minimizing the model size and maximizing the average accuracy on all the inference tasks. However, real-world IoT tasks have various service-level objectives (SLOs). Models compressed by existing methods struggle to simultaneously meet SLOs in multiple dimensions, such as latency and accuracy. In this work, we study model compression with a joint consideration of SLO awareness and task adaptation. Through our extensive experience with model compression across various IoT tasks, we observe that the importance of individual channels in contributing to accuracy is heavily influenced by task-specific data distribution. Therefore, we design a channel Shapley algorithm to estimate the importance of individual channels in DNNs and propose a deep reinforcement learning based controller to incorporate SLOs into the compression objective. Integrating these designs, we propose and prototype ChannelZip, the first SLO-aware channel compression framework. Extensive evaluations on real IoT model serving systems show the effectiveness in task adaptation of ChannelZip. ChannelZip outperforms strong model compression baselines by 3.77% accuracy and achieves a 69% average parameter compression ratio. Real-world deployment on different IoT devices shows that ChannelZip meets all task SLOs and achieves up to 2.32 × inference speedup.

Intelligent voice systems are widely utilized to control smart home applications, which raises significant privacy and security concerns. Recent studies have revealed their vulnerability to adversarial attacks, replay attacks, and so on. However, these attacks rely on the victim’s voice data. In our work, we investigate a stealthy and command-independent attack that does not necessitate collecting victims’ voices. Our proposed attack, IUAC, misleads the voice system to go against the victim’s will, regardless of the commands delivered. Our core concept is to train highly robust attack commands through the construction of diverse data, rendering the user’s commands negligible. To achieve stealthy attacks, we leverage a high-frequency carrier to construct an inaudible universal adversarial command. Extensive experiments conducted with real-world datasets demonstrate that our attack system attains an average attack success rate of 96% while resisting environmental interference. Moreover, our attack success rate against real-world voice systems is 4.52× higher than the state-of-the-art. Finally, we propose an effective defense mechanism and provide experimental tests to validate its efficacy.

In response to the escalating prevalence of inference attacks on network traffic, traffic shaping emerges as a highly effective strategy to curb the leakage of user privacy information. Although many traffic shaping mechanisms have been proposed, their privacy-preserving performance has mostly been evaluated empirically or by focusing solely on inter-event traffic shaping methods, thereby neglecting intra-event information. In this work, we develop a general framework considering intra-event characteristics for quantifying privacy leakage. The proposed evaluation framework provides quantitative assessments of privacy risks and can guide the design and improvement of traffic-shaping mechanisms. Our theoretical results identify the limitation of existing approaches, namely, the absence of a flexible trade-off between privacy-preserving effectiveness and cost. This trade-off is critical for IoT applications, as devices are highly heterogeneous in terms of resources (i.e., bandwidth) and need to meet service-level objectives (e.g., latency). To achieve this design goal, we model the trade-off problem as multi-armed bandits and propose an online traffic-shaping algorithm named DPShaping. DPShaping has a proven privacy-preserving guarantee and supports flexible trade-offs between effectiveness and communication cost. We compare our approach with three representative algorithms. Experimental results show that compared with state-of-the-art schemes, DPShaping achieves lower attack accuracy (only 16.8%) and reduces bandwidth and delay.

In recent years, technologies such as Artificial Intelligence have been of great help in driving smart production in industry. However, some links with high complexity and flexibility still rely on manual labor. This paper focuses on the design process of electrical equipment. Based on the realistic needs of enterprises, we design a real-time system for automated recognition and analysis of blueprints of electrical equipment, AI-Blueprint. It automatically extracts multi-modal information from the user’s blueprints, comprehends it, and quickly matches it with the best-standardized schemes in the enterprise database. On average, each blueprint is processed in as little as ${1 . 2 3}$ seconds, which significantly reduces labor costs. We carefully designed several specific modules to build our system, including the Document Element Detector, Diagram Extractor, Table Extractor, and Feature Extractor. To achieve good performance and fast response of AI-Blueprint, we proposed some innovative algorithms and tested their impact on the system performance under different parameters. Numerous experiments have shown that AI-Blueprint can accurately understand information such as circuit diagrams and tables in electrical equipment blueprints, and recognize and classify them precisely and quickly.

With the rapid development of internet and computer technologies, video has become increasingly prevalent in information dissemination and daily communication. However, video data often contains a substantial amount of sensitive and private information, leading to security risks during its usage, such as privacy breaches, copyright infringements, and data theft. Current efforts in video encryption primarily focus on block-level or whole-frame-level encryption within video frames, which fails to achieve fine-grained privacy protection and lacks personalized access control. Moreover, many approaches also suffer from inadequate real-time performance and excessive storage overhead.To address these issues, we proposes a personalized pixel-level scheme for video privacy protection: PPL-enc. PPL-enc employs an instance segmentation model to delineate regions containing privacy information at the pixel level within video frames. Subsequently, we apply a highly real-time quadruple encryption algorithm to the pixel values within sensitive regions, which is performed before encoding and is robust to lossy compression. Different encryption keys utilize Attribute-Based Encryption (ABE) for personalized access control, allowing terminal users with different identities to access different content upon video decryption. We conduct extensive evaluations on the quality and efficiency of encryption and decryption, as well as security analysis, to demonstrate the efficiency of our proposed scheme. Experimental results show that the single-frame encryption overhead of PPL-enc is consistently less than 0.1s across videos of various resolutions, with the encrypted area having an average PSNR of 8.50 dB and an average SSIM of 0.079.

Online video-centric service is an emerging application paradigm that enables users to access personalized video services using public equipment. However, it also brings many privacy and security issues, since the online videos might be accessed by different users. To protect the video content privacy against untrusted recipients, we need to take fine-grained control of access permissions to video content. The same video content might be accessible to certain recipients while being restricted to others. Traditional methods generate and encode multiple redacted versions of the same video, leading to substantial increases in storage, processing, and communication costs, which is difficult in adapting to the demands of the ubiquitous multimedia era. Enabling cost-effective, personalized, and fine-grained access control for video content presents a significant challenge.Video inpainting methods have gained popularity for their notable ability to remove objects with plausible pixels. In this work, we introduce the Object-grained Video Inpainting (OVI) framework for personalized access control – objects in videos are accessible only to authorized users and are visually coherently blocked for all others. OVI is efficient irrespective of user count. We implement the prototype and evaluate its performance via security study, reconstruction effectiveness, and efficiency. The experimental results show that OVI speeds up video sharing and reduces communication savings by a Θ(n) factor over the baselines when there are n different accessing groups.

To accurately understand engineering drawings, it is essential to establish the correspondence between images and their description tables within the drawings. Existing document understanding methods predominantly focus on text as the main modality, which is not suitable for documents containing substantial image information. In the field of visual relation detection, the structure of the task inherently limits its capacity to assess relationships among all entity pairs in the drawings. To address this issue, we propose a vision-based relation detection model, named ViRED, to identify the associations between tables and circuits in electrical engineering drawings. Our model mainly consists of three parts: a vision encoder, an object encoder, and a relation decoder. We implement ViRED using PyTorch to evaluate its performance. To validate the efficacy of ViRED, we conduct a series of experiments. The experimental results indicate that, within the engineering drawing dataset, our approach attained an accuracy of 96 % in the task of relation prediction, marking a substantial improvement over existing methodologies. The results also show that ViRED can inference at a fast speed even when there are numerous objects in a single engineering drawing.

With the development of the Internet of Things and artificial intelligence, the deployment and inference of intelligent models have gradually raised concerns. To reduce the huge computation and storage overhead of modern deep neural networks, many studies use model pruning techniques to reduce the model size and computational cost. However, existing pruning techniques usually require model fine-tuning, which incurs high additional overhead, making them difficult to apply to real-world scenarios. In this work, we focus on vision model compression and present F2Zip, a scenario-adaptive finetuning-free pruning framework for embedded devices. First, we propose a scenario complexity measurement that quantifies scenario changes with pixel-level entropy. By analyzing the scenario complexity, F2Zip adaptively evaluates the importance of different channels and layers of the model using only a small amount (tens) of unlabeled data. Then we design a multi-constraint knapsack solver to prune scenario-unrelated redundant channels. We implemented and deployed F2Zip in surveillance scenarios and tested different models on videos collected from both public and real-world sources. Experimental results show that F2Zip is free of model fine-tuning in various scenarios. F2Zip reduces the end-to-end deployment time by 89.8% and reduces energy cost by 79.5%, which shows that F2Zip is computationally friendly for embedded devices. Without fine-tuning and any accuracy degradation, F2Zip achieves up to 50.2% parameter reduction, outperforming baseline methods by 35.1%.

The ubiquitous data collection has raised potential risks of leaking physical and private attribute information associated with individuals in a collected dataset. A data collector who wants to collect data for provisioning its machine learning (ML)-based services requires establishing a privacy-preserving data collection protocol for data owners. In this work, we design, implement, and evaluate a novel privacy-preserving data collection protocol. Specifically, we validate the functionality of the data collection protocol on behalf of data owners. First, the ML-based services are not always predefined, it is challenging for a data collector to combat inference of private attributes and user identity from the collected data while maintaining the utility of data. To address the challenge, we reconstruct the data by designing a data transformation model based on the autoencoder and clustering. Second, it is necessary to ensure that the reconstructed data satisfy certain privacy-preserving properties as untrusted data collectors can provide the data transformation models. Therefore, we utilize detection models and design an efficient enclave-based mechanism to validate that the reconstructed data's private attribute estimation probability is bounded by the predefined thresholds. Extensive experiments demonstrate our protocol's effectiveness, such as significantly reducing the accuracy of private attribute detection

As one of the most commonly used AIoT sensors, smart cameras and their supporting services, namely cloud video surveillance (CVS) systems, have brought great convenience to people’s lives. Recent CVS providers use different machine learning techniques to improve their services (regarded as tasks) based on the uploaded video. However, uploading data to the CVS providers may cause severe privacy issues. Existing works that remove privacy information could not achieve a high tradeoff between data usability and privacy, because the importance of information varies with the task. In addition, it is challenging to design a real-time privacy protection mechanism, especially in resource-constrained smart cameras. In this work, we design a task-driven and efficient video privacy protection mechanism for a better tradeoff between privacy and data usability. We use Class Activation Mapping to protect privacy while preserving data usability. To improve the efficiency, we utilize the motion vector and residual matrix produced during video codec. Our work outperforms the region of interest–based methods in data protection while preserving data usability. The attack accuracy drops 70%, while the task accuracy is comparable to those without protection (within ± 4%). The average protection frame rate of the High Definition video can exceed 16 fps+ even on a CPU.

End-edge collaborative inference enhances computational efficiency by segmenting a deep neural network (DNN) model into two parts, executed across the end device and the edge node. However, existing collaborative inference strategies often involve transmitting original inputs from the end device to the edge node, resulting in significant risks of user detail leakage without requiring input reconstruction. Therefore, in this work, we present SecoInfer, a secure layer-level DNN end-edge collaborative inference framework. SecoInfer achieves joint optimization of data privacy and inference latency for DNN partition solutions that meet latency constraints, supported by three key designs. First, the privacy-aware DNN layer projection measurement quantifies the difficulty adversaries encounter in reconstructing the original input from the intermediate output of each layer. Then, the latency-privacy integrated structure modeling enables the direct calculation of the privacy measurement and inference latency for each partition solution from a list element or a directed acyclic graph (DAG) cut. Finally, the two-stage latency constraint adjustment scheme narrows down the search space of feasible partition solutions at the block level and fine-tunes the final one to meet the latency constraint based on layer depth. We prototype SecoInfer, utilizing a Raspberry Pi 4B as the end device and a server with an NVIDIA GeForce RTX 3060 GPU as the edge node. Experimental results demonstrate that under latency constraints of 20 ms, 33 ms, and 40 ms, SecoInfer reduces adversarial data reconstruction by 9.84%, 19.26%, and 25.18%, respectively, without any loss of task model accuracy. SecoInfer also enhances efficiency, reducing the time needed to determine optimal end-edge partition solutions on a Raspberry Pi 4B by 18.04%.

User authentication is essential for threat defense and data protection. Existed authentication systems have some known limitations, such as spoofing attacks, privacy leakage, and user-unfriendliness. In this paper, we propose BackLip, a novel anti-spoofing authentication system based on lip reading. We employ Wi-Fi backscatter-based technology to recognize users lip reading due to its various advantages, e.g. privacy protection, low power consumption, and low cost. Our system is touch-free and passphrase-independent, making it user-friendly, especially for the elderly and disabled. We first filter out irrelevant interference and enhance the signal-to-noise ratio of lip-reading signals by modulating the backscatter tags and constructing a series of suitable filters. Then, we study the energy distribution and steady-state characteristics of the backscattered signal caused by lip-reading movement at different frequencies to extract passphrase-independen lip-reading fingerprints. We build a theoretical model to analyze and prove the feasibility of our method and design an adaptive correction method to resist the interference caused by distance and angle changes. Additionally, we propose an adaptive segmentation algorithm to label lip-reading motions automatically. Extensive experiments demonstrate that BackLip has an average accuracy of 92.1% and is improved to 96.5% when users use the same passphrases.

With the increasing number of deep learning applications, the optimization of deep learning model performance has become a central focus of research. One of the critical indicators is model-inference latency. Rapid and accurate prediction of this latency is essential for effective model design and deployment. However, existing methods, which rely on performance data, often yield inaccurate predictions. To tackle this challenge, we propose a model-inference latency predictor specifically designed for mobile devices. Our predictor can rapidly and accurately predict the inference latency of various Convolutional Neural Network (CNN) models under various load conditions of the device. The key idea behind our predictor is to combine hardware features, model computational graph features, and convolution latency features to achieve more precise latency prediction. We comprehensively evaluate our system on mobile devices. The accuracy of our predictor within ±10% error exceeds 90% at different frequencies, CPU utilizations, and bandwidths. Additionally, our predictor demonstrates the capability to predict model-inference latency under various conditions, which is not possible with existing methods. As a result, our system can offer precise and fast model-inference latency for various CNN models and devices.

By combining the advantages of Federated Learning (FL) and Split Learning (SL), SplitFed Learning (SFL) has become a widely applied scheme to train deep neural networks (DNNs) in distributed training scenarios with high data privacy requirements. However, SFL systems also meet severe security challenges caused by the distributed architecture, while current attacking methods mainly focusing on security vulnerabilities but neglecting the enforceability of conducting attacks. In this paper, we study the scenario-oriented feature poison attack for the first time that aims at interfering the features achieved by the distributed clients in SFL during the transmission process. Specifically, we propose a Supvirus attacker alone with a Supvirus GAN to achieve the attack. The Supvirus attacker and Supvirus GAN conduct online attacks on the transmission link without directly affecting the system software and hardware, and based on the GAN structure, it can better focus on scene changes and obtain better optimization strategies. Experiment results on an established SFL system confirm strong stealthiness, low deployment and implementation costs, and a high attack success rate of our proposed attacker and the GAN. Our approach does not require extra modifications on the clients or the server in SFL. Meanwhile, the accuracy drop by our approach achieves 57.35% at most and the drop can still reach 42.37% in situations where only a single client is attacked. The attack time requirement, attacker convergence performance, and the differences in features before and after attacks remain similar with the normal clients.

Existing intelligent piano learning systems mainly assist the player by camera, which generally only consider the fingering that can only reflect the performance problem from a limited perspective. Thus, we propose PianoWatch, a multi-dimensional assistance system based on wrist wearable devices. PianoWatch extracts more accurate patterns by analyzing data from microphone, camera, accelerometers and gyroscopes. Then it gives corresponding playing advices, in addition to pitch, including fingering, depth and even mood, through an evaluation model. We implement the prototype of PianoWatch and evaluate it by 20 volunteers. Extensive experiments show it can achieve 93% F1-Score on the playing pattern extraction, and more than 95% of users would like to try it to assist their piano learning.

Recent studies show that smart home devices are vulnerable to passive network observers, referred to as adversaries. An adversary can use mobile devices as a sniffer to infer device events (e.g., a door is opened/closed) even through encrypted WiFi traffic. However, to obtain high event identification accuracy, existing works heavily rely on either machine learning (ML) or large traffic feature sizes, which results in a costly retraining or pairing process. In this paper, we generate an event traffic fingerprint and propose an event identification method with good extensibility. To address the interference of network fluctuations and unrelated traffic, we filter all redundant and event-unrelated packets and extract event packet sequences based on time interval. After processing, we generate a representative packet-level fingerprint for each event and identify the event based on fingerprint matching. The precision and recall of event identification reach 96.77% and 92.31% on average. Our traffic processing method can work as an add-on to existing ML-based event inference methods, which leads to at least 12% increase on the accuracy.

Nowadays, Internet of Things (IoT) based smart home system is equipped with a large number of smart devices, such as smart speakers and cameras, which can greatly facilitate users to control and automate their home environment. However, recent studies have shown that smart home system is at great risk of privacy leakage. Especially, external attackers can infer user privacy information by passively sniffing encrypted smart home network traffic. Traditional methods mainly focus on sniffing WiFi devices but pay less attention to other commodity devices such as Zigbee and Bluetooth (BLE). In this paper, we focus on inferring fine-grained sensitive details about users using diverse commodity devices. We apply deep learning techniques to infer users' behaviors through identifying and localizing smart home devices being used due to the excellent performance of deep learning in many fields. Specifically, we first pre-process encrypted device traffic, select valid features, and use Convolutional Neural Networks (CNN) for device identification. In addition, we extract the Received Signal Strength Indicator (RSSI) from the frame information of traffic packets and employ Sparse Autoencoder (SAE) to extract stable and distinguishable high-dimensional features for RSSI measurement. Features are fed into a Multilayer Perceptron (MLP) to predict the device's localization. In this way, we can infer human activity by identifying and localizing the devices being used. Extensive experiment results show that our work can achieve a mean position estimation error of 1.34m even in an unseen environment, outperforming other common- used localization algorithms based on RSSI fingerprints.

Obfuscation technologies have been well established for on-device image privacy protection, including pixelization, blurring, scribbling, sticker-covering, and inpainting. Despite their remarkable resistance to human observation, recent studies find that some of them are vulnerable to attacks by neural network-based recognition methods. In this work, we reveal the risk of privacy re-disclosure post image protection. Given an obfuscation-protected image, the privacy information includes 1) where the obfuscated region is and 2) what the hidden privacy-related objects are. Thus we focus on uncovering categories of privacy-related objects to evaluate the effectiveness of obfuscation technologies. Under severe obfuscation, unfortunately, even powerful object recognition models can hardly infer hidden privacy information.

Major cloud service providers with well-equipped infrastructure, experienced machine learning (ML) expertise, and enriched training datasets are building ML-as-a-Service (MLaaS) systems, in which clients can query ML-based prediction services with their data. Instead of moving private data to the cloud, in this work, we design, implement, and evaluate a novel secure ML system to enable MLaaS on edge devices. To protect the proprietary ML models on edge devices from revealing to the clients while maintaining a real-time inference is challenging. Existing privacy-preserving ML techniques can hardly satisfy real-time requirements. In our solution, we employ a secure enclave (e.g., SGX) to offer security and provide better efficiency than cryptographic techniques. However, the enclave alone cannot achieve real-time capability due to its limited capacity. We observe that the ML model imposes a severe accuracy degradation when adding noise to a few model weights. Based on this, we design a suite of novel solutions to optimize the performance of secure enclave-based inference service at the edge by enclosing only$1\%$computation within secure enclaves. Our work can achieve up to a$7.8\times$increase in efficiency and a$27\times$reduction in memory usage compared to the state-of-the-art.

Network function virtualization is a key enabling technology in future wireless networks for flexible and efficient sharing of network resources. Due to the increasing heterogeneity of network resource providers, a blockchain-based distributed architecture is a promising solution to enable reliable and transparent virtualized network function (VNF) management. However, since on-chain storage and computation are costive, it becomes a challenging task to achieve efficient VNF management with blockchain. In this paper, we first introduce a consortium blockchain for collaborative VNF management among network resource providers. Then, we propose an authenticated VNF dictionary that can be stored as a succinct authenticator on blockchain to support rich VNF query functionalities and efficient verifications of query results. Moreover, we design a dictionary pruning strategy to securely generate a compact authenticator for a given query, which reduces unnecessary memory accesses of the original dictionary when VNF queries are represented as arithmetic circuits. Finally, we conduct extensive experiments with a consortium blockchain network. The experimental results demonstrate that our pruning strategy is efficient for both on-chain and off-chain VNF management.

Voice input users’ speech recordings are being collected by service providers and shared with third parties, who may abuse users’ voiceprints, identify them by voice, and learn their sensitive speech content. In this work, we designSpeech Sanitizerto perturb users’ speech recordings so that the sanitized speech can be safely shared with third parties. First, we desensitize speech content by identifying sensitive words, localizing them in the audio using DTW-based keyword spotting, and substituting them with safe words. Both common and personalized sensitive words are identified and replaced. Then, we anonymize users’ voiceprints with a carefully designed voice conversion mechanism that is resistant to de-anonymization attacks. Meanwhile, we try to preserve the utility of the sanitized speech, measured by the accuracy of speech recognition performed on it. We implement Speech Sanitizer and present extensive experimental results that validate the effectiveness and efficiency of our algorithms. It is demonstrated that we are able to reduce the chance of a user's voice being identified from 50 people by 83.7 percent while keeping the drop of speech recognition accuracy within 19.1 percent. We can also easily relax the privacy level to improve speech recognition accuracy.

Privacy has become one of the major concerns in cloud video surveillance. Privacy protection of the surveillance videos strive to protect users' privacy information without hampering regular security tasks of the surveillance, meanwhile retains the system's high accuracy and efficiency. The current state of the art in protecting the video privacy is mainly realized through Privacy Region Protection, which only protects the privacy regions while keeps the non-privacy regions visually intact so that processing in the cloud is still feasible. However, the problem of determining the privacy regions has been ignored and not properly addressed. In this paper, we propose a novel notion - concept graph, and with the aid of that, we develop our system - PatronuS to determine the privacy regions subject to satisfying both privacy and security requirements. We further propose an event distilling model and a privacy inference model to assist in determining specific privacy regions. And we evaluate PatronuS in real-world settings and demonstrate its efficiency in privacy protection without degrading system's surveillance functionality.

Machine learning (ML) has shown its impressive performance in the modern world, and many corporations leverage the technique of machine learning to improve their service quality, e.g., Facebook's DeepFace. Machine learning models with a collection of private data being processed by a training algorithm are deemed to be increasingly confidential. Confidential models are typically trained in a centralized cloud server but publicly accessible. ML-as-a-service (MLaaS) system is one of running examples, where users are allowed to access trained models and are charged on a pay-per-query basis.

Sign language is a natural and fully-formed communication method for deaf or hearing-impaired people. Unfortunately, most of the state-of-the-art sign recognition technologies are limited by either high energy consumption or expensive device costs and have a difficult time providing a real-time service in a daily-life environment. Inspired by previous works on motion detection with wearable devices, we propose Sign Speaker - a real-time, robust, and user-friendly American sign language recognition (ASLR) system with affordable and portable commodity mobile devices. SignSpeaker is deployed on a smartwatch along with a smartphone; the smartwatch collects the sign signals and the smartphone outputs translation through an inbuilt loudspeaker. We implement a prototype system and run a series of experiments that demonstrate the promising performance of our system. For example, the average translation time is approximately $1.1$ seconds for a sentence with eleven words. The average detection ratio and reliability of sign recognition are 99.2% and 99.5%, respectively. The average word error rate of continuous sentence recognition is 1.04% on average.

We propose a novel motion sensing/recognition system, called DF-Mose, which marries low-power wireless backscattering and device-free sensing in one clean sheet. DF-Mose is an accurate, interference tolerable motion-recognition system that counts repetitive motions without using scenario-dependent templates or profiles within 5% count error and enables multiuser to perform certain motions simultaneously based on the nature of backscattered signals and dedicated signal separation method. With little efforts in learning the patterns, our method could achieve 95.2% motion-recognition accuracy for a variety of 7 typical motions.

In this paper, a set of accountable protocols denoted as AccountTrade is proposed for big data trading among dishonest consumers. For achieving a secure big data trading environment, AccountTrade achieves book-keeping ability and accountability against dishonest consumers throughout the trading (i.e., buying and selling) of datasets. We investigate the consumers' responsibilities in the dataset trading, then we design AccountTrade to achieve accountability against dishonest consumers that are likely to deviate from the responsibilities. Specifically, a uniqueness index is defined and proposed, which is a new rigorous measurement of the data uniqueness for this purpose. Furthermore, several accountable trading protocols are presented to enable data brokers to blame the misbehaving entities when misbehavior is detected. The accountability of AccountTrade is formally defined, proved, and evaluated by an automatic verification tool as well as extensive simulation with real-world datasets. Our evaluation shows that AccountTrade incurs at most 10-kB storage overhead per file, and it is capable of 8-1000 concurrent data upload requests per server.

Coflow is gradually prevalent as a new traffic structure in data centers, which allows applications to convey their application-level semantics into the network. Meanwhile, optical circuit switches (OCS) are increasingly deployed in data centers due to the superiority in hardware, such as high bandwidth and low energy consumption. However, few works in the literature considered both coflow scheduling with OCS. In this paper, we study the coflow scheduling problem in an OCS-based data center network, with an aim to minimize the coflow completion time (CCT). We derive an online algorithm called OMCO to solve this problem. OMCO can not only optimize the circuit utilization but also minimize the number of circuit reconfigurations. Extensive simulations with real-world data traces show that OMCO outperforms other existing solutions dramatically. Compared with a FIFO-based scheme and the shortest-coflow-first heuristic, OMCO reduces the average coflow completion time by up to 61% and 62%, respectively.

Privacy-preserving data publishing has been a heated research topic in the last decade. Numerous ingenious attacks on users' privacy and defensive measures have been proposed for the sharing of various data, varying from relational data, social network data, spatiotemporal data, to images and videos. Speech data publishing, however, is still untouched in the literature. To fill this gap, we study the privacy risk in speech data publishing and explore the possibilities of performing data sanitization to achieve privacy protection while preserving data utility simultaneously. We formulate this optimization problem in a general fashion and present thorough quantifications of privacy and utility. We analyze the sophisticated impacts of possible sanitization methods on privacy and utility, and also design a novel method - key term perturbation for speech content sanitization. A heuristic algorithm is proposed to personalize the sanitization for speakers to restrict their privacy leak (p-leak limit) while minimizing the utility loss. The simulations of linkage attacks and sanitization on real datasets validate the necessity and feasibility of this work.

Deep Learning has shown promising performance in a variety of pattern recognition tasks owning to large quantities of training data and complex structures of neural networks. However conventional deep neural network (DNN) training involves centrally collecting and storing the training data, and then centrally training the neural network, which raises much privacy concerns for the data producers. In this paper, we study how to enable deep learning without disclosing individual data to the DNN trainer. We analyze the risks in conventional deep learning training, then propose a novel idea - Crowdlearning, which decentralizes the heavy- load training procedure and deploys the training into a crowd of computation-restricted mobile devices who generate the training data. Finally, we propose SliceNet, which ensures mobile devices can afford the computation cost and simultaneously minimize the total communication cost. The combination of Crowdlearning and SliceNet ensures the sensitive data generated by mobile devices never leave the devices, and the training procedure will hardly disclose any inferable contents. We numerically simulate our prototype of SliceNet which crowdlearns an accurate DNN for image classification, and demonstrate the high performance, acceptable calculation and communication cost, satisfiable privacy protection, and preferable convergence rate, on the benchmark DNN structure and dataset.

We are speeding toward a not-too-distant future when we can perform human-computer interaction using solely our voice. Speech recognition is the key technology that powers voice input, and it is usually outsourced to the cloud for the best performance. However, user privacy is at risk because voiceprints are directly exposed to the cloud, which gives rise to security issues such as spoof attacks on speaker authentication systems. Additionally, it may cause privacy issues as well, for instance, the speech content could be abused for user profiling. To address this unexplored problem, we propose to add an intermediary between users and the cloud, named VoiceMask, to anonymize speech data before sending it to the cloud for speech recognition. It aims to mitigate the security and privacy risks by concealing voiceprints from the cloud. VoiceMask is built upon voice conversion but is much more than that; it is resistant to two de-anonymization attacks and satisfies differential privacy. It performs anonymization in resource-limited mobile devices while still maintaining the usability of the cloud-based voice input service. We implement VoiceMask on Android and present extensive experimental results. The evaluation substantiates the efficacy of VoiceMask, e.g., it is able to reduce the chance of a user's voice being identified from 50 people by a mean of 84%, while reducing voice input accuracy no more than 14.2%.

We consider a private data set composed of a set of individuals, and the data are outsourced to a remote cloud server. We revisit the classic query auditing problem in this outsourcing scenario; the cloud audits each newly arrived query on a single attribute, and the query is rejected if answering it compromises any individual's privacy. Various query auditing issues have been studied and addressed before. However, previous auditing schemes either have the difficulty of removing denial threats, or lack the analysis of utility (which is defined as the number of answered queries). In this paper, we study the auditing of a sequence of polynomial-time computable queries. Each query is of format f(X̃) a, where f is any polynomial function, X̃ is a subset of the private data set, and the answer is either “yes” or “no”. Existing methods cannot be applied directly to audit such a query, because it intermingles several types of functions (e.g., sum and max/min). Hence, we propose CASTLE, which is an inequality query auditing scheme that evaluates the risk of answering a query based on the query history and determines whether a newly arrived query should be answered correctly against a denial threat. Furthermore, to overcome the limitations of the existing query auditing mechanisms, which are of low utility, we relax CASTLE to increase the utility by returning answers with slight perturbations. We show that our method can be applied to audit intermingled equality queries with an extension. Experiments are conducted to evaluate the efficiency and effectiveness of our methods.

We propose AccountTrade, a set of accountable protocols, for big data trading among dishonest consumers. To secure the big data trading environment, our protocols achieve book-keeping ability and accountability against dishonest consumers who may misbehave throughout the dataset transactions. Specifically, we study the responsibilities of the consumers in the dataset trading and design AccountTrade to achieve accountability against the dishonest consumers who may try to deviate from their responsibilities. Specifically, we propose uniqueness index, a new rigorous measurement of the data uniqueness, as well as several accountable trading protocols to enable data brokers to blame the dishonest consumer when misbehavior is detected. We formally define, prove, and evaluate the accountability of our protocols by an automatic verification tool as well as extensive evaluation in real-world datasets. Our evaluation shows that AccountTrade incurs negligible constant storage overhead per file (<;10KB), and it is able to handle 8-1000 concurrent data uploading per server depending on the data types.

Sign Language is a natural and fully-fledged communication method for deaf and hearing-impaired people. In this demo, we propose the first SmartWatch-based American sign language (ASL) recognition system, which is more comfortable, portable and user-friendly and offers accessibility anytime, anywhere. This system is based on the intuitive idea that each sign has its specific motion pattern which can be transformed into unique gyroscope and accelerometer signals and then analyzed and learned by using Long-Short term memory recurrent neural network (LSTM-RNN) trained with connectionist temporal classification (CTC). In this way, signs and context information can be correctly recognized based on an off-the-shelf device (eg. SmartWatch, Smartphone). The experiments show that, in the Known user split task, our system reaches an average word error rate of 7.29% to recognize 73 sentences formed by 103 ASL signs and achieves detection ratio up to 93.7% for a single sign. The result also shows our system has a good adaptation, even including new users, it can achieve an average word error rate of 21.6% at the sentence level and reach an average detection ratio of 79.4%. Moreover, our system performs real time ASL translation, outputting the speech within 1.69 seconds for a sentence of 12 signs in average.

This paper presents a framework for privacy-preserving video delivery system to fulfill users' privacy demands. The proposed framework leverages the inference channels in sensitive behavior prediction and object tracking in a video surveillance system for the sequence privacy protection. For such a goal, we need to capture different pieces of evidence which are used to infer the identity. The temporal, spatial and context features are extracted from the surveillance video as the observations to perceive the privacy demands and their correlations. Taking advantage of quantifying various evidence and utility, we let users subscribe videos with a viewer-dependent pattern. We implement a prototype system for off-line and on-line requirements in two typical monitoring scenarios to construct extensive experiments. The evaluation results show that our system can efficiently satisfy users' privacy demands while saving over 25% more video information compared to traditional video privacy protection schemes.

Recently a number of systems have been developed to implement and improve the visual communication over screen-camera links. In this paper we study an opposite problem: how to prevent unauthorized users from videotaping a video played on a screen, such as in a theater, while do not affect the viewing experience of legitimate audiences. We propose and develop a light-weight hardware-free system, called Kaleido, that ensures these properties by taking advantage of the limited disparities between the screen-eye channel and the screen-camera channel. Kaleido does not require any extra hardware and is purely based on re-encoding the original video frame into multiple frames used for displaying. We extensively test our system Kaleido using a variety of smartphone cameras. Our experiments confirm that Kaleido preserves the high-quality screen-eye channel while reducing the secondary screen-camera channel quality significantly.