Abstract 3D Gaussian Splatting has markedly impacted neural rendering by achieving impressive fidelity and performance. Despite this achievement, it is not readily applicable to developing interactive applications. Real‐time applications like XR apps and games require functions such as animation, UV mapping and level of detail (LOD) simultaneously manipulated through a 3D model. To address this need, we propose a modelling strategy analogous to typical 3D models, which we call 3D Gaussian Model (3DGM). 3DGM relies on attaching 3D Gaussians on the triangles of a mesh proxy, and the key idea is to bind sheared 3D Gaussians in texture space and re‐projecting them back to world space through implicit shell mapping; this design naturally enables deformation and UV mapping via the proxy. Further, to optimize speed and fidelity based on different viewing distances, each triangle can be tessellated to change the number of involved 3D Gaussians adaptively. Application‐wise, we will show that our proxy‐based 3DGM is capable of enabling novel deformation without animated training data, texture transferring via UV mapping of the 3D Gaussians, and LOD rendering. The results indicate that our model achieves better fidelity for deformation and better optimization of fidelity and performance given different viewing distances. Further, we believe the results indicate the potential of our work for enabling interactive applications for 3D Gaussian Splatting.

Locomotion has a marked impact on user experience in VR, but currently, common to-go techniques such as steering and teleportation have their limitations. Particularly, steering is prone to cybersickness, while teleportation trades presence for mitigating cybersickness. Inspired by how we manipulate a picture on a mobile phone, we propose illumotion, an optical-illusion-based method that, we believe, can provide an alternative to these two typical techniques. Instead of zooming in a picture by pinching two fingers, we can move forward by “zooming” toward part of the 3D virtual scene with pinched hands. Not only is the proposed technique easy to use, it also seems to minimize cybersickness to some degree.illumotion relies on the manipulation of optics; as such, it requires solving motion parameters in screen space and a model of how we perceive depth. To evaluate it, a comprehensive user study with 66 users was conducted. Results show that, compared with either teleportation, steering or both, illumotion has better performance, presence, usability, user experience and cybersickness alleviation. We believe the result is a clear indication that our novel optically-driven method is a promising candidate for generalized locomotion.

Have you ever picked up a stuffed toy and pretended to play with it in your childhood? We are motivated by the novel use of stuffed toys in enhancing extended reality interaction. A key goal of extended reality is to induce the feeling of presence in its users. Naturally mapped control interface has been shown to enhance presence. The literature also indicates that a high degree of freedom tracking is important to extended reality. Based on these observations, we show that a free-form naturally mapped control interface is well-motivated via a theoretical contextualization. We explore the possibility of building such a controller in the form of stuffed toys. To realize stuffed toys as controllers, a novel soft-pose estimator empowered by cage-based deformation is proposed. It is shown to be effective in tracking the poses and deformations of real soft objects even by training with synthetic data only. Three gameplay prototypes are developed to demonstrate that interactive play can be enabled by the soft-pose estimator. They also form the basis for two user studies that validate the success of tracking stuffed toys with the soft-pose estimator for interactive play.

In recent years, there is a growing phenomenon with children playing mobile devices. Overplaying mobile devices, however, may not lead to a balanced childhood. Lack of physical touch-and-feel stimulus is detrimental to children. Can touch-and-feel stuffed toys be made more appealing to children for a wider variety on their playtime diet? We explore a stuffed toy controller which integrates elements from both digital and make-believe games in an attempt to enrich children interaction experience. A user study was conducted to evaluate the effectiveness of the proposed controller while another children-oriented user study shows that make-believe augmented reality games played by the toy controller can enhance children’s interest in stuffed toys.

Games have grown to be an important part of our society and naturally, its incorporation with education has become a rising trend Together with the strong need for STEM talents, this paper proposed that STEM sandbox game-based learning video games could enhance students' interest and performance in the four disciplines. It is further argued that simulation realism, creativity and interactivity are three important components for this purpose. To promote their development, Planetary Marching Cubes (PMC), a development basis for STEM games has been discussed, along with a game developed with it, HelloPlanet. Students in primary and secondary schools have been invited to play HelloPlanet, and one of the main contributions of this paper is to demonstrate the improvement in both students' interest and performance in STEM. Students and teachers feedback are both very positive, implying that the education sector is fully ready to integrate game-based learning. When comparing with other methods of planet generation, most students picked PMC. This consolidates the argument of the importance of the three components, while also demonstrates the potential of PMC as a development basis for educational sandbox games.