Ruwen Zhang, Bo Liu, Zhang Sheng Xiang, Yida Chen, Hantao Zhao, Ding Ding, Jiahui Jin, Jiuxin Cao. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Federated instruction tuning of large language models (LLMs) has recently emerged as a promising research direction for preserving data privacy while enabling collaborative model adaptation. However, due to the heterogeneity of local instruction data across clients in federated settings, assigning the same trainable parameter size to all clients may compromise local learning effectiveness and limit the overall performance of the global model. To address this challenge, we propose FedTPA, a dynamic pruning-based strategy that allocates and adjusts the adapter dimensions of local models based on the distribution of local instruction data and trends in training loss. This allows the trainable parameter size on each client to better align with the complexity and characteristics of its local data. We evaluate FedTPA across multilingual, multi-task, and varying degrees of data heterogeneity scenarios. Experimental results demonstrate that FedTPA outperforms existing federated instruction tuning methods, achieving up to a 3% improvement in Rouge-L scores.

Dementia, one of the leading causes of neurodegenerative mortality in older adults, remains widely misunderstood by the general public—not only stigmatized socially, but also subject to persistent misconceptions about its symptoms, progression, and lived experience. These misunderstandings hinder timely care, empathy, and social support. To address this, we introduce LetheVR, a first-person serious game designed to promote both empathic understanding of individuals living with dementia and cognitive awareness of the disease itself. Targeted at general audiences, the system adopts experiential methods embedded within a game-based structure. Unlike traditional media and static educational tools, LetheVR integrates immersive symptom simulation, narrative-driven gameplay, and guided pedagogical reflection to engage users in the lived experience of dementia. In a controlled study with 60 participants, LetheVR significantly outperformed conventional interventions in improving measured empathy levels and symptom understanding. These findings highlight the potential of Virtual Reality combining with serious games and experiential methods as effective public health interventions for reshaping attitudes and correcting public misunderstandings about dementia.

Serving as a fundamental interaction in Virtual Reality, Locomotion technology defines how users navigate and explore the immersive virtual environments with high degree of freedom. Accurate prediction of locomotion not only enhances the sense of presence and ease of movement in virtual environment but also benefits VR applications through context-aware optimization such as pre-rendering and scene streaming. To leverage the superior understanding and causal modeling capabilities of Foundation Models (FMs) in the domain of numerical prediction, we apply FMs to time-series data, enabling more accurate estimation of users’ future spatial coordinates based on historical motion and gaze data. In this article, we introduce LoCoFoMo, an Adapter-based FM architecture specifically designed for future trajectory prediction in VR contexts. We conduct extensive experiments to evaluate the effectiveness of LoCoFoMo and its components. The proposed model demonstrates strong competitiveness when compared with several trajectory prediction and temporal reasoning models, evidenced by a performance gain of over 25% against the best baseline on datasets with interaction paradigms like Touchpad and Arm-Swing, coupled with superior stability in mitigating error accumulation.

The general-purpose Segment Anything Model (SAM) is limited by the inherent constraints of RGB sensors, which render it inadequate for challenging real-world scenarios such as adverse lighting conditions and rapid motion. In contrast, event cameras, a novel type of bio-inspired visual sensor, offer distinct imaging advantages, including high temporal resolution and a high dynamic range. The event streams generated by these cameras provide spatiotemporal dynamic cues that are often absent in conventional image frames. To overcome the limitations of RGB-based models, we propose SAM with Event-based Assistance (EvSAM) , a novel RGB-event multi-modal semantic segmentation framework. EvSAM leverages the strong generalization capabilities of SAM while incorporating the complementary characteristics of event data to enhance scene comprehension, particularly under adverse conditions. To address the challenges of fusing two modals (image and event) with large data format discrepancy, we introduce two core components: the Multi-spatiotemporal-scale Patch Alignment Block (MS 2 PAB) and the Event-based Feature Injector (EFInj) for SAM. Specifically, the MS \({}^{2}\) PAB captures spatiotemporal semantic coherence from the event stream and transforms it into a frame-based complementary representation using a multi-spatiotemporal alignment strategy. The EFInj introduces a dynamic event feature update mechanism, wherein the fused features at a given layer guide the adaptive generation of deeper event representations. This process facilitates the integration of RGB spatial semantics with event-based motion cues. Owing to these core designs, EvSAM demonstrates superior performance on event-based semantic segmentation datasets, thereby fully validating its distinct advantages in handling extreme visual scenarios. Furthermore, we extend our model to the task of depth estimation, which further demonstrates its strong generalization ability and scalability for various downstream applications.

We propose a novel task, Virtual Object Grounding (VOG) . It aims to predict plausible locations in an image for inserting virtual objects that align with a given textual description. This VOG task can address the challenge of providing region constraints for object insertion in image editing, thereby ensuring the consistency of irrelevant areas in the image. To support this task, we construct Virtual Segmentation (VirtualSeg) dataset, a dataset of over 92,000 samples automatically generated from VrR-VG via a four-step dataset construction pipeline. This pipeline employs CLIP to automatically filter out low-quality data samples, ensuring the quality of VirtualSeg. Furthermore, we propose the VirLLaVA model, a novel VOG framework built upon LLaVA-7B. By equipping the MLLM backbone with two sequences of learnable tokens and a dual grounding module, and by guiding the model during training to learn step-by-step how to locate virtual objects, our method enables it to reason about their positions from textual and visual inputs. Experiments show that VirLLaVA significantly improves performance in VOG, while also offering a promising direction for consistent and automated image editing. The code and dataset are available at https://github.com/Royxia0818/MLLM_for_VOG .

Despite the increasing prevalence of online social interaction, challenges such as insufficient immersion and lack of interactivity still persist. To overcome these limitations, this study developed a multi-user virtual reality (VR) cinema system with motion capture (Mocap) and multi-user VR technology. The system demonstrated strengths in overcoming physical space restrictions and saving travel costs, providing a more enriched interactive experience and fulfilling social needs under special circumstances, and facilitating metaverse applications. Furthermore, to give insight into the further design of multi-user VR cinemas, this study investigated the impact of virtual humans' (VHs') characteristics on user perception and experience of bilateral virtual companions, by evaluating the impact of movement and voice realism on immersion, social presence and intimacy. The results show that while neither movement nor voice realism significantly influences immersion, voice realism rather than movement realism significantly affects social presence and intimacy.

Locomotion is a fundamental interaction in Virtual Reality (VR). Current locomotion methods, such as redirected walking, walking-in-place, and teleportation, make use of limited physical space and interaction mapping. However, there remains significant potential for improvement, particularly in reducing equipment burden and enhancing immersion. To locate these limitations, we rethink the procedure of VR walking interaction through the Human Information Processing paradigm. Finding that the peripherals' requirements and potential conflict in artificially designed interaction mappings are the bottlenecks in bridging intention and action, we developed MyGO, an AI-assisted locomotion prediction method. MyGO predicts users' future trajectories from their subtle movements, collected only by a VR headset, using a multitask learning (MTL) model. The proposed model demonstrates competitive results in both dataset validation and real-time studies. The code is available at https://github.com/ZichengLiu-seu/basic-MyGO.

Virtual reality (VR), as an immersive interactive technology that offers visual and auditory experiences, creates a fertile ground for both art appreciation and creation. However, current VR art face challenges such as limited experiences, restricted interaction and inadequate engagement. Accordingly, this study explores the design of the emotional feedback mechanism in museum-based VR environments. We proposed a novel paradigm for art interaction, embedding emotional feedback into the artistic re-creation to enrich and personalize the experience of artwork. Through an empirical study$(\mathrm{N}=48)$with a between-subjects design, we demonstrates that this artistic recreation enhances user engagement and emotional states by emphasizing audiovisual quality, interactive flexibility, and narrative coherence with the physical museum. We also shed light on the future design of interaction between people and art that aim to facilitate deeper, more engaged experience.

With metaverse attracting increasing attention from both academic and industry, the application of virtual reality (VR) has extended beyond 3D immersive viewing/gaming to a broader range of areas, such as banking, shopping, tourism, education, and so on, which involves a growing amount of sensitive and private user data into VR systems. However, with current password-based user authentication schemes in mainstream VR devices, studies demonstrate that side-channel attacks can pose a severe threat to VR user privacy. To mitigate the threat, we propose a novel panoramic-image-based VR user authentication system, i.e., Pivot , to defend against such attacks, yet maintain high usability. Specifically, in Pivot , we design an image-random-pivoting-based user interaction mechanism to assist users in quickly and securely selecting memorable points of interest in a panoramic image. Then an image region segmentation algorithm is designed to automatically scatter the points to regions to form the customized graphic password for the user, which could ensure a sufficiently large password space and also reduce the near-region point misclicks. Afterward, the region indexes are used to generate the hashed password for authentication. Both theoretical security analysis and extensive user studies demonstrate that Pivot is secure and user-friendly in practice.

Social interactions form an essential aspect of people’s life, however, it is quite challenging for individuals to handle a wide range of social situations. Therefore, a variety of training systems have been developed to improve their skills. This literature review seeks to give an overview of the state of the art of technology-supported systems for social skills training. The studies eligible for inclusion described a technology-supported system with the purpose of training social skills and included an experimental or observational study to evaluate the efficacy of the system. 225 studies (224 publications) with 216 systems were identified, characterized, and analyzed in this literature review. Using the taxonomy as put forward in this study, the analysis shows that the majority of these systems were screen-based applications, with virtual reality technology being the most frequently observed. The systems most often targeted communication skills that focus on transferring information to produce greater understanding, i.e. mending general communication impairments in children with autism. In terms of functions, support for learning-by-doing was the most observed function, while focusing on job interviews provided the largest number of functions. Finally, the studies reported overwhelmingly positively regarding the systems’ impact, including 76 studies with a randomized controlled trial design. Still, most studies only used a quasi-experimental design based on self-report measures. We anticipate the proposed taxonomy to be a starting point for researchers to position their work and that the review will help them with gaining inspiration for the design and evaluation of social skills training systems.

Recent advancements in federated learning have shown promising results in resource-constrained edge environments. However, with mobile devices becoming more capable of collecting data, individual client models are unable to utilize the available data due to their devices’ limited support for complex model training. Conversely, non-portable devices possess substantial computational resources, but the data they autonomously collect is insufficient to support the training of complex models. In this paper, we introduce HASFL, a novel split federated learning (SFL) framework that supports model structure heterogeneity across devices and decouples computation from the model. Through circular group training, HASFL enables mobile devices to utilize complex models to train their own data while ensuring that non-portable devices harness the data collected by mobile users. HASFL effectively addresses the challenges of applying advanced machine learning models in resource-constrained environments, leveraging the collective power of distributed devices without compromising data security. We implemented a circular group allocation method using the online algorithm to ensure cooperative training among heterogeneous models within each group while minimizing training time. In addition, we have conducted experiments to evaluate the performance of HASFL on various datasets and model architectures and analyzed the communication overhead of HASFL. The experimental results demonstrate that HASFL supports the training of heterogeneous models and significantly enhances the model’s accuracy with a relatively small increase in communication overhead.

While Virtual Reality Perspective-Taking (VRPT) demonstrates its efficiency in inducing empathy, its application primarily focuses on vulnerable humans, not animals. Existing animal-related works mainly targets farm animals and wildlife. In this work, we focus on stray animals and introduce iStrayPaws, a VRPT system that simulates stray animals’ challenging lives. The system offers users an immersive first-person journey into the world of stray animals encountering different difficulties like inclement weather, hunger, and illnesses. Enriched with audio-visual and kinesthetic design, the system seeks to deepen users’ understanding of stray animals’ life and foster profound emotional connections. To evaluate the system, a user study was conducted, which showed that VRPT recipients exhibited significant improvement in both state and trait empathy compared to traditional method. Our research not only delivers a novel, accessible, and interactive animal empathy experience but also provides innovative solutions for addressing stray animal issues and advancing broader animal welfare work.

Previous multi-modal transformers for RGB-D salient object detection (SOD) generally directly connect all patches from two modalities to model cross-modal correlation and perform multi-modal combination without differentiation, which can lead to confusing and inefficient fusion. Instead, we disentangle the cross-modal complementarity from two views to reduce cross-modal fusion ambiguity: 1) Context disentanglement. We argue that modeling long-range dependencies across modalities as done before is uninformative due to the severe modality gap. Differently, we propose to disentangle the cross-modal complementary contexts to intra-modal self-attention to explore global complementary understanding, and spatial-aligned inter-modal attention to capture local cross-modal correlations, respectively. 2) Representation disentanglement. Unlike previous undifferentiated combination of cross-modal representations, we find that cross-modal cues complement each other by enhancing common discriminative regions and mutually supplement modal-specific highlights. On top of this, we divide the tokens into consistent and private ones in the channel dimension to disentangle the multi-modal integration path and explicitly boost two complementary ways. By progressively propagate this strategy across layers, the proposed Disentangled Feature Pyramid module (DFP) enables informative cross-modal cross-level integration and better fusion adaptivity. Comprehensive experiments on a large variety of public datasets verify the efficacy of our context and representation disentanglement and the consistent improvement over state-of-the-art models. Additionally, our cross-modal attention hierarchy can be plug-and-play for different backbone architectures (both transformer and CNN) and downstream tasks, and experiments on a CNN-based model and RGB-D semantic segmentation verify this generalization ability.

Sandplay therapy is a novel psychological consultation and treatment method for people suffering from mental stress and psychological issues. It is popular among the masses owing to its interest and curative effect. However, traditional sandplay games face several problems, such as the requirement of large space, low accessibility, difficulty in keeping treatment records, and lack of engagement. To solve these problems, we put forward a sandplay system based on virtual reality technology. This system provides a virtual scenario of the sandplay room, allowing users to place various sandplay miniatures naturally and smoothly into the sandbox with multiple themes by using motion capture gloves. What is more, users can switch to the first-person perspective to experience the world they have built. Additionally, the system could collect users’ eye-tracking data and analyze it by a machine learning algorithm to estimate individuals’ emotions which could assist in further treatment. Finally, twelve participants were invited to experience the system. They were asked to evaluate this system as well as their emotional status before and after using it. The experimental results indicate that the participants are satisfied with this system, while their emotions can be improved to a certain extent.

Depression is a severe mental illness that can lead to negative moods and activities. The traditional clinical approach for diagnosing depression is face-to-face consultation, which is limited by time and space. Virtual Reality (VR), as a novel technology with higher accessibility and lower cost, can serve as an effective digital approach to diagnosing psychological disorders. In VR systems, users are exposed to various experimental scenarios, gaining immersive and interactive experiences. Recent research has demonstrated a relationship between depression and low spatial memory navigation ability (SMNA). Based on these considerations, we propose a VR system to detect one’s depression level by measuring spatial memory navigation performances. The system consists of three virtual scenarios with different spatial scales and dimensions. To study the system’s effectiveness, a pilot study with eight participants was conducted. The results showed differences in the participants’ spatial memory navigation performances in the three scenarios and a correlation between depression level and their spatial memory navigation performances.

The mental health of older adults is a vital issue in the era of global aging. Empathy towards older adults constitutes a crucial component of social interaction, as it engenders awareness of the physical and mental obstacles encountered by this population. Empathy promotes individuals to be more friendly, considerate, and prosocial when interacting with older adults on various occasions, such as volunteering and professional caring. Various approaches have been used to promote people's empathy, but disadvantages like low participation enthusiasm, high cost, and time-consuming cannot be ignored and are not easy to deal with. Aimed at facilitating people's empathy towards older adults, we developed EmpathiaVR, a virtual reality older adults experience simulation system. It provides a multi-sensory mixed reality experience, including vision, hearing, and kinesthesis, from an older person's perspective, which is beneficial for provoking users' empathy towards older adults. To investigate the effectiveness of the system, an empirical study was conducted with 24 participants. The experiment employed a between-subjects design with two groups. The experiment results from both the subjective reports and behavioral variables indicated that the system enhanced people's empathy.

Effective psychological interventions for anxiety disorders often include exposure to fearful situations. However, individuals with low self-efficacy may find such exposure too overwhelming. We created a vicarious experience in virtual reality, which enables observation of one’s experience from a first person perspective without actual performance and which might increase self-efficacy. With similarities to both traditional vicarious experiences and direct experiences, the level of self-identification with the experience was hypothesized to affect self-efficacy and its relationship with direct experiences. To test this, vicarious experiences with two distinct levels of self-identification were compared in a between-subjects experiment (n=60). After being exposed to a vicarious experience of giving lectures on elementary arithmetic in front of a virtual audience with either a high or low level of self-identification with the public speaker, participants from both conditions actively gave another lecture. The results revealed that self-identification affected people’s self-efficacy after vicarious experience. They further revealed that self-identification is a moderator of (1) the correlation between perceived performance and self-efficacy, (2) the correlation between self-efficacy measured after the vicarious and the follow-up direct experience; and (3) the correlation between the sense of presence reported in the vicarious and in the follow-up direct experience. We anticipate that the first-person-perspective experiences with high-level of self-identification have the potential to be beneficial for training where changing people’s self-efficacy is desirable.

Abstract Virtual cognitions (VCs) are a stream of simulated thoughts people hear while emerged in a virtual environment, e.g. by hearing a simulated inner voice presented as a voice over. They can enhance people’s self-efficacy and knowledge about, for example, social interactions as previous studies have shown. Ownership and plausibility of these VCs are regarded as important for their effect, and enhancing both might, therefore, be beneficial. A potential strategy for achieving this is the synchronization of the VCs with people’s eye fixation using eye-tracking technology embedded in a head-mounted display. Hence, this paper tests this idea in the context of a pre-therapy for spider and snake phobia to examine the ability to guide people’s eye fixation. An experiment with 24 participants was conducted using a within-subjects design. Each participant was exposed to two conditions: one where the VCs were adapted to eye gaze of the participant and the other where they were not adapted, i.e. the control condition. The findings of a Bayesian analysis suggest that credibly more ownership was reported and more eye-gaze shift behaviour was observed in the eye-gaze-adapted condition than in the control condition. Compared to the alternative of no or negative mediation, the findings also give some more credibility to the hypothesis that ownership, at least partly, positively mediates the effect eye-gaze-adapted VCs have on eye-gaze shift behaviour. Only weak support was found for plausibility as a mediator. These findings help improve insight into how VCs affect people.

With the rapid development and popularity of cloud computing technology, more and more Collaborative Virtual Environment (CVE) systems are migrated to cloud computing environment to improve the effectiveness of resource usage. Virtual Machine (VM) placement in cloud data center is a key issue of providing high-efficient cloud platform for CVE system. However, most existing VM placement algorithms ignore the following characteristics of actual cloud environment: (1) VMs deployment requests arrive and leave dynamically, (2) Cloud data center usually consists of many heterogeneous Physical Machines (PMs). Ignoring these two characteristics result in an inefficient and unbalanced use of multiple resources of PMs. Thus using these algorithms directly will lead to a poor resource utilization. In this article, we propose a two-phase online VM placement algorithm, which helps the cloud data center to minimize different resource usages and aims at a more efficient use of multiple resources. Our algorithm selects the most suitable PM type for VM based on Cosine Similarity, and adaptively maps VMs to PMs by using an approximation algorithm. The proposed algorithm is evaluated by simulations. Experimental results show our proposed algorithm ensures a more efficient use of multiple resources over the existing approaches.