User identification is essential for securing electronic devices, especially in immersive environments such as virtual and augmented reality (VR/AR). Traditional identification methods typically rely on proactive, one-time authentication, rendering them susceptible to common security threats and inadequate for providing continuous protection after login. Furthermore, existing user identification solutions for VR/AR head-mounted displays (HMDs) often compromise user convenience or require additional hardware, increasing cost and complexity. Although significant research has explored biometric-based identification in VR/AR, many proposed methods face practical challenges: they often depend on extensive sensor data or demand intrusive user interactions, limiting their suitability for diverse real-world scenarios. In this work, we present a lightweight and practical biometric user identification algorithm that utilizes only head movement patterns—eliminating the need for auxiliary sensors such as external cameras or controller-based inputs, and imposing no extra tasks on the user. Our approach is platform-independent and can generalize to new tasks without retraining. We validate its effectiveness through comprehensive experiments on multiple public datasets. Results demonstrate that our method achieves high identification accuracy with minimal data and computational overhead, making it a scalable and viable solution for continuous user identification in next-generation AR/VR applications.

Federated Learning (FL) offers significant advancements in user/data privacy, learning quality, model efficiency, scalability, and network communication latency. However, it faces notable security challenges, particularly with the emergence of backdoor attacks. The distributed nature of FL complicates the development of backdoor-resistant systems compared to traditional machine learning environments. In this paper, we propose a novel approach to turn the perceived curse of model inversion (MI) attacks into a blessing, using them as a tool for detecting backdoor attacks in FL environments. Leveraging MI outputs, we propose a K-means-based feature extraction and Isolation-Forest-based anomaly detection algorithm to analyze behavior and detect abnormal learning performance, thereby identifying backdoor attacks. Experimental results demonstrate the effectiveness and superior performance of our method in detecting backdoor attacks within FL systems.

Virtual Reality (VR) and Augmented Reality (AR) have witnessed a surge in popularity, revolutionizing various industries and enhancing user experiences. As these technologies continue to evolve, ensuring secure user identification becomes increasingly important. However, existing identification methods often come with vulnerabilities or require costly hardware implementations. To address these challenges, we propose a novel semi-biometric information-based user identification approach leveraging multimodal techniques. By analyzing the user’s viewing patterns and gaze behavior in the runtime, we can accurately identify individuals. This approach offers a promising solution for seamless user identification in VR and AR applications, without compromising on user experience or requiring expensive hardware modifications. To verify the efficiency of our proposed algorithm, we test our algorithm using a public dataset, the result of which shows a high classification accuracy. Additionally, beyond traditional evaluation metrics, we perform a transferability test, demonstrating the adaptability of our solution to new scenarios with robust generality.

Modern computer vision and AI algorithms have become highly effective in analyzing high-dimensional image and video content for various tasks. Recently, some have exploited the power of computer vision to develop cheating tools for video games, which pose a serious threat to the gaming community and the game industry. Using human pose estimation algorithms, these cheating tools can assist players by automatically targeting and shooting with very high precision and accuracy. Compared to classic cheating methods, these tools are generally much more undetectable as they can mimic real “competent” players. To counter this threat, we propose a machine learning-based approach that leverages the concept of adversarial attacks to generate perturbations that fool such human pose estimation algorithms, preventing cheaters from gaining unfair advantages. In this work, we first implement the video game cheating systems and then we propose and implement our solution. In the end, we use the cheating system to evaluate the efficiency of our proposed algorithms in defending against such cheating tools. Experimental results show that our algorithm can effectively deceive advanced human pose estimation algorithms by adding invisible perturbations to the characters in the video game, maintaining a fair and healthy gaming environment.

Video games is one of the most popular multimedia forms and generate higher profits than the traditional film industry. In the meantime, with the advances of deep learning, computer vision algorithms have become more powerful for analyzing the video content and have been applied in the FPS video games as an advanced cheating tools, which have taken the video games industry by storm. Such algorithms, including the object detection and human pose estimations, could analyze and understand the video content in each frame and further help the player to automatically identify and aim at the enemies with extremely fast reaction. Compared to the classic cheating tools, computer-vision-based cheating tools are harder to detect and defend against because they do not need to manipulate the software or the system but purely simulate how a well trained and skilled human gamer plays the video game. In this paper, we propose a proactive and comprehensive defense approach, which generates perturbations that are not perceptible to humans yet can still mislead the computer vision algorithms. More specifically, this comprehensive approach includes two parts, the defense approach aims to fail the computer vision-based cheating tools to detect the in-game characters while the penalty approach aims to fool the computer vision-based cheating tools to detect the fake regions as in-game characters, which not only worsen the cheating experience but also serve as a trigger for detecting the cheating behavior. In this work, we first implement the object detection based cheating tools as the evaluation environment. Then, we implement our proposed defense, penalty and comprehensive approaches and evaluate the performance with four popular video games. The results show that our comprehensive approach obtains a high success rate with minor impact to user experience quality.

3D video surveillance has become the new trend in security monitoring with the popularity of 3D depth cameras in the consumer market. While enabling more fruitful surveillance features, the finer-grained 3D videos being captured would raise new security concerns that have not been addressed by existing research. This paper explores the security implications of live 3D surveillance videos in triggering biometrics-related attacks, such as face ID spoofing. We demonstrate that the state-of-the-art face authentication systems can be effectively compromised by the 3D face models presented in the surveillance video. Then, to defend against such face spoofing attacks, we propose to proactively and benignly inject adversarial perturbations to the surveillance video in real time, prior to the exposure to potential adversaries. Such dynamically generated perturbations can prevent the face models from being exploited to bypass deep learning-based face authentications while maintaining the required quality and functionality of the 3D video surveillance. We evaluate the proposed perturbation generation approach on both an RGB-D dataset and a 3D video dataset, which justifies its effective security protection, low quality degradation, and real-time performance.

This paper aims to address the significant power challenges in live virtual reality (VR) streaming (a.k.a., 360-degree video streaming), where the VR view rendering and the advanced deep learning operations (e.g., super-resolution) consume a considerable amount of power draining the battery-constrained VR headset. We develop EdgeVR, a power optimization technique for live VR streaming, which offloads the on-device VR rendering and deep learning operations to an edge server for power savings. To address the significantly increased motion-to-photon (MtoP) latency due to the edge offloading, we develop a live VR viewport prediction method to pre-render the VR views on the edge server and compensate for the round-trip delays. We evaluate the effectiveness of EdgeVR using an end-to-end live VR streaming system with an empirical VR head movement dataset involving 48 users watching 9 VR videos. The results reveal that EdgeVR achieves power-efficient live VR streaming with low MtoP latency.

Deep neural networks (DNNs) have been deployed on FPGAs to achieve improved performance, power efficiency, and design flexibility. However, the FPGA-based DNNs are vulnerable to fault injection attacks that aim to compromise the original functionality. The existing defense methods either duplicate the models and check the consistency of the results at runtime, or strengthen the robustness of the models by adding additional neurons. However, these existing methods could introduce huge overhead or require retraining the models. In this paper, we develop a runtime verification method, namely Siamese path verification (SPV), to detect fault injection attacks for FPGA-based DNN execution. By leveraging the computing features of the DNN and designing the weight parameters, SPV adds neurons to check the integrity of the model without impacting the original functionality and, therefore, model retraining is not required. We evaluate the proposed SPV approach on Xilinx Virtex-7 FPGA using the MNIST dataset. The evaluation results show that SPV achieves the security goal with low overhead.

Deep neural networks (DNNs) have demonstrated phenomenal success in image classification applications and are widely adopted in multimedia internet of things (IoT) use cases, such as smart home systems. To compensate for the limited resources on the IoT devices, the computation-intensive image classification tasks are often offloaded to remote cloud services. However, the offloading-based image classification could pose significant security and privacy concerns to the user data and the DNN model, leading to effective adversarial attacks that compromise the classification accuracy. The existing defense methods either impact the original functionality or result in high computation or model re-training overhead. In this paper, we develop a novel defense approach, namely Fake Gradient, to protect the privacy of the data and defend against adversarial attacks based on encryption of the output. Fake Gradient can hide the real output information by generating fake classes and further mislead the adversarial perturbation generation based on fake gradient knowledge, which helps maintain a high classification accuracy on the perturbed data. Our evaluations using ImageNet and 7 popular DNN models indicate that Fake Gradient is effective in protecting the privacy and defending against adversarial attacks targeting image classification applications.

Virtual reality (VR) streaming can provide immersive video viewing experience to the end users but with huge bandwidth consumption. Recent research has adopted selective streaming to address the bandwidth challenge, which predicts and streams the user's viewport of interest with high quality and the other portions of the video with low quality. However, the existing viewport prediction mechanisms mainly target the video-on-demand (VOD) scenario relying on historical video and user trace data to build the prediction model. The community still lacks an effective viewport prediction approach to support live VR streaming, the most engaging and popular VR streaming experience. We develop a region of interest (ROI)-based viewport prediction approach, namely LiveROI, for live VR streaming. LiveROI employs an action recognition algorithm to analyze the video content and uses the analysis results as the basis of viewport prediction. To eliminate the need of historical video/user data, LiveROI employs adaptive user preference modeling and word embedding to dynamically select the video viewport at runtime based on the user head orientation. We evaluate LiveROI with 12 VR videos viewed by 48 users obtained from a public VR head movement dataset. The results show that LiveROI achieves high prediction accuracy and significant bandwidth savings with real-time processing to support live VR streaming.

Virtual reality (VR) video streaming (a.k.a., 360-degree video streaming) has been gaining popularity recently as a new form of multimedia providing the users with immersive viewing experience. However, the high volume of data for the 360-degree video frames creates significant bandwidth challenges. Research efforts have been made to reduce the bandwidth consumption by predicting and selectively streaming the user's viewports. However, the existing approaches require historical user or video data and cannot be applied to live streaming, the most attractive VR streaming scenario. We develop a live viewport prediction mechanism, namely LiveObj, by detecting the objects in the video based on their semantics. The detected objects are then tracked to infer the user's viewport in real time by employing a reinforcement learning algorithm. Our evaluations based on 48 users watching 10 VR videos demonstrate high prediction accuracy and significant bandwidth savings obtained by LiveObj. Also, LiveObj achieves real-time performance with low processing delays, meeting the requirement of live VR streaming.

Volumetric video (VV) streaming has drawn an increasing amount of interests recently with the rapid advancements in consumer VR/AR devices and the relevant multimedia and graphics research. While the resource and performance challenges in volumetric video streaming have been actively investigated by the multimedia community, the potential security and privacy concerns with this new type of multimedia have not been studied. We for the first time identify an effective threat model that extracts 3D face models from volumetric videos and compromises face ID-based authentications To defend against such attack, we develop a novel volumetric video security mechanism, namely VVSec, which makes benign use of adversarial perturbations to obfuscate the security and privacy-sensitive 3D face models. Such obfuscation ensures that the 3D models cannot be exploited to bypass deep learning-based face authentications. Meanwhile, the injected perturbations are not perceivable by the end-users, maintaining the original quality of experience in volumetric video streaming. We evaluate VVSec using two datasets, including a set of frames extracted from an empirical volumetric video and a public RGB-D face image dataset. Our evaluation results demonstrate the effectiveness of both the proposed attack and defense mechanisms in volumetric video streaming.

Live virtual reality (VR) streaming has become a popular and trending video application in the consumer market providing users with 360-degree, immersive viewing experiences. To provide premium quality of experience, VR streaming faces unique challenges due to the significantly increased bandwidth consumption. To address the bandwidth challenge, VR video viewport prediction has been proposed as a viable solution, which predicts and streams only the user’s viewport of interest with high quality to the VR device. However, most of the existing viewport prediction approaches target only the video-on-demand (VOD) use cases, requiring offline processing of the historical video and/or user data that are not available in the live streaming scenario. In this work, we develop a novel viewport prediction approach for live VR streaming, which only requires video content and user data in the current viewing session. To address the challenges of insufficient training data and real-time processing, we propose a live VR-specific deep learning mechanism, namely LiveDeep, to create the online viewport prediction model and conduct real-time inference. LiveDeep employs a hybrid approach to address the unique challenges in live VR streaming, involving (1) an alternate online data collection, labeling, training, and inference schedule with controlled feedback loop to accommodate for the sparse training data; and (2) a mixture of hybrid neural network models to accommodate for the inaccuracy caused by a single model. We evaluate LiveDeep using 48 users and 14 VR videos of various types obtained from a public VR user head movement dataset. The results indicate around 90% prediction accuracy, around 40% bandwidth savings, and premium processing time, which meets the bandwidth and real-time requirements of live VR streaming.

Heterogeneous CPU-FPGA systems have been shown to achieve significant performance gains in domain-specific computing. However, contrary to the huge efforts invested on the performance acceleration, the community has not yet investigated the security consequences due to incorporating FPGA into the traditional CPU-based architecture. In fact, the interplay between CPU and FPGA in such a heterogeneous system may introduce brand new attack surfaces if not well controlled. We propose a hardware isolation-based secure architecture, namely HISA, to mitigate the identified new threats. HISA extends the CPU-based hardware isolation primitive to the heterogeneous FPGA components and achieves security guarantees by enforcing two types of security policies in the isolated secure environment, namely the access control policy and the output verification policy. We evaluate HISA using four reference FPGA IP cores together with a variety of reference security policies targeting representative CPU-FPGA attacks. Our implementation and experiments on real hardware prove that HISA is an effective security complement to the existing CPU-only and FPGA-only secure architectures.