In this paper, we present a neural sampling method, EllipssianNet, which predicts 2D Gaussians from an input RGB image. Trained with a Voronoi diagram-based synthetic dataset, EllipssianNet outputs a center map and a covariance map, which are combined with the colors sampled from the input image to generate 2D Gaussians. The Gaussians are anisotropic and aligned with local complexities of the input RGB image. The 2D Gaussians are converted into 3D ones that are then optimized and rasterized in the 3D Gaussian Splatting framework. EllipssianNet is tested in two applications. In Gaussian-based image representation, initialization with EllipssianNet enables faster convergence and higher rendering quality. EllipssianNet is also seamlessly integrated into a real-time SLAM system, producing high-quality reconstructions under online constraints. The code and trained weight are available at https://github.com/medialab-ku/EllipssianNet

We propose an XR (eXtended Reality) system that simulates and visualizes airflow and thermal comfort controlled by an air conditioner. This system reconstructs an indoor space to make its 3D map, tracks the air conditioner’s pose without markers, runs fluid and thermal comfort simulators, and visualizes the simulation results on a mobile XR device. It enables users to instantly find comfortable/uncomfortable spots in the indoor space and control the air conditioner more effectively. The modules for 3D mapping, object pose tracking, simulation and visualization are integrated on a single XR device, making the system portable and widely usable.

ABSTRACT In this article, we propose an AR system that facilitates a user's natural interaction with virtual objects in an augmented reality environment. The system consists of three modules: human pose and shape estimation, camera‐space calibration, and physics simulation. The first module estimates a user's 3D pose and shape from a single RGB video stream, thereby reducing the system setup cost and broadening potential applications. The camera‐space calibration module estimates the user's camera‐space position to align the user with the input RGB image. The physics simulation enables seamless and physically natural interaction with virtual objects. Two prototyping applications built upon the system prove an enhancement in the quality of interaction, fostering a more immersive and intuitive user experience.

In this paper, we propose to integrate superpixel-guided sampling into the framework of 3D Gaussian Splatting for novel view synthesis. Given a sequence of frames, where each frame is a pair of RGB-D image and the camera pose that captures the image, we first select the keyframes. Then, we decompose each keyframe image into superpixels, back-project each superpixel’s center into 3D space, and initialize a 3D Gaussian at the back-projected position. This superpixel-guided sampling produces a set of sparse but well-distributed Gaussians, which enables the optimization procedure to converge quickly. The experimental results show that with a significantly reduced computing cost, we can synthesize a novel view the quality of which is comparable to that generated by the state-of-the-art methods.

This paper presents a mobile AR system for visualizing airflow and temperature change made by virtual air conditioners. Even though there have been efforts to integrate the results of airflow/temperature simulation into the real world via AR, they support neither interactive modeling of the environments nor real-time simulation. This paper presents an AR system, where 3D mapping and air conditioner installation are made interactively, and then airflow/temperature simulation and visualization are made at real time. The proposed system is designed in a client-server architecture, where the server is in charge of simulation and the rest is taken by the client.

In the context of mid-air manipulation, this paper presents the effects of allowing the user to dynamically switch between 1DOF, 2DOF, and 3DOF operations. Such “manipulation with switchable DOF” has been widely used in commercial graphics packages and is increasingly being adopted for virtual reality editing, where the 3D scene is constructed through mid-air interaction in immersive virtual environments. However, its effectiveness and advantages/disadvantages have not been investigated. This paper compares “manipulation with switchable DOF” with “manipulations with DOF separation,” which allows only 1DOF operations, and “manipulation without DOF separation,” which provides 3DOF operations. Using translation, rotation, and scaling, three methods were evaluated in terms of completion time and precision. The experiment results showed that “manipulation with switchable DOF” outperformed “manipulation with DOF separation” in terms of completion time whereas three methods were comparable in terms of precision. “Manipulation with switchable DOF” was further analyzed, and the results showed that the more 3DOF operations led to the shorter completion time and the more 1DOF operations led to the higher precision.

The reduced gravity experienced in lunar or Martian surfaces can be simulated on the earth using a cable-driven system, where the cable lifts a person to reduce his or her weight. This paper presents a novel cable-driven system designed for the purpose. It is integrated with a head-mounted display and a motion capture system. Focusing on jump motion within the system, this paper proposes to scale the jump and reports the experiments made for quantifying the extent to which a jump can be scaled without the discrepancy between physical and virtual jumps being noticed by the user. With the tolerable range of scaling computed from these experiments, an application named retargeted jump is developed, where a user can jump up onto virtual objects while physically jumping in the real-world flat floor. The core techniques presented in this paper can be extended to develop extreme-sport simulators such as parasailing and skydiving.