The convergence of artificial intelligence (AI) and extended reality (XR) technologies (AIXR) promises innovative applications across many domains. However, the sensitive nature of data (e.g., eye-tracking) used in these systems also raises significant privacy concerns, as adversaries can exploit this data and these models to infer personal information. Prior research has primarily examined membership inference attacks (MIA) to leak privacy at the model-level and re-identification attacks (RDA) at the dataset-level, separately as individual attacks. While these attacks are relevant to the XR domain, launching these attacks as individual attacks is not practical and incurs more attack cost. To address this gap, we present the first comprehensive study of chain of privacy (CHOP) attacks against AIXR applications. We demonstrate how adversaries can launch such attacks with a high success rate, in a cost-effective way, by sequentially combining MIA and Attribute inference attacks (AIA) to re-identify XR users without access to raw XR data, training distributions, or model parameters. We evaluate our proposed method in realistic AIXR settings by adopting deep learning (DL)-based cybersickness detection as a representative AIXR application. Specifically, we train two state-of-the-art DL models on two open-source datasets: Simulation 2021 and VRWalking, and a new XR cybersickness dataset constructed from 34 participants via a user study. Our findings reveal that the proposed CHOP attacks pose severe risks to DL-based cybersickness detection, achieving re-identification rates of up to 94% and 97% on the open-source and the developed cross-linked datasets, respectively, underscoring the feasibility and severity of cross-dataset privacy violations. Furthermore, cost analysis reveals that the proposed CHOP attack is ≈ 2× more cost-effective than traditional individual attacks for re-identifying XR users. Finally, we propose two ε-differential privacy (DP)-enabled privacy-preserving mechanisms: Differentially Private Stochastic Gradient Descent (DPSGD) and Private Aggregation of Teacher Ensembles (PATE) to mitigate CHOP attacks. Our results show that the proposed defense reduces the re-identification rate by up to 88% and 79% while maintaining high model utility, with classification accuracies of up to 94% and 92% for the same datasets using Transformer models.

The convergence of artificial intelligence (AI) and extended reality (XR) technologies (AI XR) promises innovative applications across many domains. However, the sensitive nature of data (e.g., eyetracking) used in these systems raises significant privacy concerns, as adversaries can exploit these data and models to infer and leak personal information through membership inference attacks (MIA) and re-identification (RDA) with a high success rate. Researchers have proposed various techniques to mitigate such privacy attacks, including differential privacy (DP). However, AI XR datasets often contain numerous features, and applying DP uniformly can introduce unnecessary noise to less relevant features, degrade model accuracy, and increase inference time, limiting real-time XR deployment. Motivated by this, we propose a novel framework combining explainable AI (XAI) and DP-enabled privacy-preserving mechanisms to defend against privacy attacks. Specifically, we leverage post-hoc explanations to identify the most influential features in AI XR models and selectively apply DP to those features during inference. We evaluate our XAI-guided DP approach on three state-of-the-art AI XR models and three datasets: cybersickness, emotion, and activity classification. Our results show that the proposed method reduces MIA and RDA success rates by up to 43 % and 39 %, respectively, for cybersickness tasks while preserving model utility with up to 97 % accuracy using Transformer models. Furthermore, it improves inference time by up to$\approx 2 \times$compared to traditional DP approaches. To demonstrate practicality, we deploy the XAI-guided DP AI XR models on an HTC VIVE Pro headset and develop a user interface (UI), namely PrivateXR, allowing users to adjust privacy levels (e.g., low, medium, high) while receiving real-time task predictions, protecting user privacy during XR gameplay. Finally, we validate our approach through a user study, which confirms that participants found the PrivateXR UI effective, with satisfactory utility and user experience.

Immersive virtual reality (VR) environments pose significant cognitive and physical challenges as users engage in multitasking scenarios involving attention management and working memory, often leading to increased cognitive load, sensory conflicts, and cybersickness, diminishing users’ performance and immersion. While traditional machine learning (ML) and deep learning (DL) methods have been employed to predict individual factors such as cybersickness or attention, they often fail to capture the interconnected and dynamic nature of these cognitive and physiological demands. Moreover, these methods typically require large volumes of labeled data, extended training times, and struggle to generalize across diverse VR contexts. To address these limitations, we propose an innovative method for predicting multiple tasks, i.e., cybersickness, cognitive load, working memory, and attention by leveraging the knowledge of pre-trained large foundation models, namely TimeGPT and Chronos. We apply two learning mechanisms, zero-shot and few-shot learning, for adapting these foundation models for multiple task predictions. We validate our approach on the open-source VRWalking dataset, utilizing multimodal data fusion and participant-specific grouping (based on age and gender), and compare it against traditional DL-based methods trained from scratch. Results show that our few-shot-based fine-tuned TimeGPT and Chronos models significantly outperform traditional DL models in multiple tasks. Specifically, the fine-tuned TimeGPT model achieves significantly lower RMSE values for predicting cybersickness, cognitive physical load, cognitive mental load, working memory, attention success rate, and reaction time, respectively, outperforming the traditional transformer. Furthermore, the fine-tuned TimeGPT model achieves a 4.52 × reduction in training time compared to a conventional Transformer model for the same prediction tasks. Moreover, we deploy the fine-tuned TimeGPT model on the HTC VIVE Pro VR headset, enabling real-time prediction of multiple task severity levels from streaming VR simulation data during gameplay.

The synergy between virtual reality (VR) and artificial intelligence (AI), specifically deep learning (DL)-based cybersickness detection models, has ushered in unprecedented advancements in immersive experiences by automatically detecting cybersickness severity and adaptively various mitigation techniques, offering a smooth and comfortable VR experience. While this DL-enabled cybersickness detection method provides promising solutions for enhancing user experiences, it also introduces new risks since these models are vulnerable to adversarial attacks; a small perturbation of the input data that is visually undetectable to human observers can fool the cybersickness detection model and trigger unexpected mitigation, thus disrupting user immersive experiences (UIX) and even posing safety risks. In this paper, we present a new type of VR attack, specifically a cybersickness attack, which successfully prevents the triggering of cybersickness mitigation by deceiving DL-based cybersickness detection models and significantly hinders the UIX. Next, we propose a novel explainable artificial intelligence (XAI)-guided cybersickness attack detection framework to detect such attacks in VR, ensuring UIX and a comfortable VR experience. We evaluate the proposed attack and detection framework using two state-of-the-art open-source VR cybersickness datasets: the Simulation 2021 dataset and the Gameplay dataset. Finally, to verify the effectiveness of our proposed method, we implement the attack and the XAI-based detection using a custom-built testbed with a VR roller coaster simulation, utilizing an HTC Vive Pro Eye headset, and conduct a user study. Our study shows that such an attack can dramatically hinder the UIX. However, our proposed XAI-guided cybersickness attack detection can successfully detect cybersickness attacks and trigger the proper mitigation, effectively reducing VR cybersickness.

Cybersickness is a common problem that users often encounter during virtual reality (VR) experiences. Several automated methods exist based on machine learning (ML)/deep learning (DL) to detect cybersickness. However, the sensitive nature of data used by these ML/DL models (e.g., eye-tracking, head-tracking, etc.) introduces significant privacy risks since adversaries could exploit this data to infer and leak sensitive personal information, track individuals, or manipulate user experiences. Our research seeks to address this gap, underscoring the necessity for a private approach to cybersickness detection to protect user privacy and ensure a better VR experience. Thus, this paper proposes a privacy-preserving mechanism for DL-enabled cybersickness detection modeled. Specifically, we employ differential privacy (DP) to develop four private DL cybersickness detection models: long short-term memory (LSTM), grated recurrent unit (GRU), convolutional neural network (CNN), and multilayer perceptron (MLP) using Simulations 2021 and Gameplay, two open-source datasets. Our proposed models show high cybersickness detection accuracy for the proposed private cybersickness models. For instance, the private LSTM model shows the cybersickness detection accuracy of up to 92% and 91% for the Simulations 2021 and Gameplay datasets, respectively. Our experimental results also exhibit the privacy-preserving nature of private cybersickness detection. For instance, the private LSTM model reduces the membership inference attack’s success rate by up to 32% and 45% for the Simulations 2021 and Gameplay datasets compared to the baseline/non-private LSTM model for the same datasets.

Virtual Reality (VR) is quickly establishing itself in various industries, including training, education, medicine, and entertainment, in which users are frequently required to carry out multiple complex cognitive and physical activities. However, the relationship between cognitive activities, physical activities, and familiar feelings of cybersickness is not well understood and thus can be unpredictable for developers. Researchers have previously provided labeled datasets for predicting cybersickness while users are stationary, but there have been few labeled datasets on cybersickness while users are physically walking. Moreover, it is unclear how walking while cybersick will affect cognitive load, even though room-scale interaction is typical in many VR games. Thus, from 39 participants, we collected head orientation, head position, eye tracking, images, physiological readings from external sensors, and the self-reported cybersickness severity, physical load, and mental load in VR. Throughout the data collection, participants navigated mazes via real walking and performed tasks challenging their attention and working memory. To demonstrate the dataset’s utility, we conducted a case study of training classifiers in which we achieved 95% accuracy for cybersickness severity classification. The noteworthy performance of the straightforward classifiers makes this dataset ideal for future researchers to develop cybersickness detection and reduction models. To better understand the features that helped with classification, we performed SHAP(SHapley Additive exPlanations) analysis, highlighting the importance of eye tracking and physiological measures for cybersickness prediction while walking. This open dataset can allow future researchers to study the connection between cybersickness and cognitive loads and develop prediction models. This dataset will empower future VR developers to design efficient and effective Virtual Environments by improving cognitive load management and minimizing cybersickness.

Virtual reality (VR) systems are known for their susceptibility to cybersickness, which can seriously hinder users’ experience. Therefore, a plethora of recent research has proposed several automated methods based on machine learning (ML) and deep learning (DL) to detect cybersickness. However, these detection methods are perceived as computationally intensive and black-box methods. Thus, those techniques are neither trustworthy nor practical for deploying on standalone VR head-mounted displays (HMDs). This work presents an explainable artificial intelligence (XAI)-based framework VR-LENS for developing cybersickness detection ML models, explaining them, reducing their size, and deploying them in a Qualcomm Snapdragon 750G processor-based Samsung A52 device. Specifically, we first develop a novel super learning-based ensemble ML model for cybersickness detection. Next, we employ a post-hoc explanation method, such as SHapley Additive exPlanations (SHAP), Morris Sensitivity Analysis (MSA), Local Interpretable Model-Agnostic Explanations (LIME), and Partial Dependence Plot (PDP) to explain the expected results and identify the most dominant features. The super learner cybersickness model is then retrained using the identified dominant features. Our proposed method identified eye tracking, player position, and galvanic skin/heart rate response as the most dominant features for the integrated sensor, gameplay, and bio-physiological datasets. We also show that the proposed XAI-guided feature reduction significantly reduces the model training and inference time by 1.91X and 2.15X while maintaining baseline accuracy. For instance, using the integrated sensor dataset, our reduced super learner model outperforms the state-of-the-art works by classifying cybersickness into 4 classes (none, low, medium, and high) with an accuracy of and regressing (FMS 1–10) with a Root Mean Square Error (RMSE) of 0.03. Our proposed method can help researchers analyze, detect, and mitigate cybersickness in real time and deploy the super learner-based cybersickness detection model in standalone VR headsets.

Cybersickness is a common ailment associated with virtual reality (VR) user experiences. Several automated methods exist based on machine learning (ML) and deep learning (DL) to detect cyber-sickness. However, most of these cybersickness detection methods are perceived as computationally intensive and black-box methods. Thus, those techniques are neither trustworthy nor practical for deploying on standalone energy-constrained VR head-mounted devices (HMDs). In this work, we present an explainable artificial intelligence (XAI)-based framework Lite VR for cybersickness detection, explaining the model's outcome, reducing the feature dimensions, and overall computational costs. First, we develop three cybersick-ness DL models based on long-term short-term memory (LSTM), gated recurrent unit (GRU), and multilayer perceptron (MLP). Then, we employed a post-hoc explanation, such as SHapley Additive Explanations (SHAP), to explain the results and extract the most dominant features of cybersickness. Finally, we retrain the DL models with the reduced number of features. Our results show that eye-tracking features are the most dominant for cybersickness detection. Furthermore, based on the XAI-based feature ranking and dimensionality reduction, we significantly reduce the model's size by up to 4.3×, training time by up to 5.6×, and its inference time by up to 3.8×, with higher cybersickness detection accuracy and low regression error (i.e., on Fast Motion Scale (FMS)). Our proposed lite LSTM model obtained an accuracy of 94% in classifying cyber-sickness and regressing (i.e., FMS 1–10) with a Root Mean Square Error (RMSE) of 0.30, which outperforms the state-of-the-art. Our proposed Lite VR framework can help researchers and practitioners analyze, detect, and deploy their DL-based cybersickness detection models in standalone VR HMDs.

Cybersickness can be characterized by nausea, vertigo, headache, eye strain, and other discomforts when using virtual reality (VR) systems. The previously reported machine learning (ML) and deep learning (DL) algorithms for detecting (classification) and predicting (regression) VR cybersickness use black-box models; thus, they lack explainability. Moreover, VR sensors generate a massive amount of data, resulting in complex and large models. Therefore, having inherent explainability in cybersickness detection models can significantly improve the model’s trustworthiness and provide insight into why and how the ML/DL model amved at a specific decision. To address this issue, we present three explainable machine learning (xML) models to detect and predict cybersickness: 1) explainable boosting machine (EBM), 2) decision tree (DT), and 3) logistic regression (LR). We evaluate xML-based models with publicly available physiological and gameplay datasets for cybersickness. The results show that the EBM can detect cybersickness with an accuracy of 99.75% and 94.10% for the physiological and gameplay datasets, respectively. On the other hand, while predicting the cybersickness, EBM resulted in a Root Mean Square Error (RMSE) of 0.071 for the physiological dataset and 0.27 for the gameplay dataset. Furthermore, the EBM-based global explanation reveals exposure length, rotation, and acceleration as key features causing cybersickness in the gameplay dataset. In contrast, galvanic skin responses and heart rate are most significant in the physiological dataset. Our results also suggest that EBM-based local explanation can identify cybersickness-causing factors for individual samples. We believe the proposed xML-based cybersickness detection method can help future researchers understand, analyze, and design simpler cybersickness detection and reduction models.