We present a framework that captures sparse Lagrangian flow information from a volume of real liquid and reconstructs its detailed kinematic information in real time. Our framework can perform flow reconstruction even when the liquid is disturbed by an object of unknown movement and shape. Through a large dataset of liquid moving under external disturbance, an agent is trained using reinforcement learning to reproduce the target flow kinematics with only the captured sparse information as inputs while remaining oblivious to the movement and the shape of the disturbance sources. To ensure that the underlying simulation model faithfully obeys physical reality, we also optimize the viscosity parameters in Smoothed Particle Hydrodynamics (SPH) using classical fluid dynamics knowledge and gradient-based optimization. By quantitatively comparing the reconstruction results against real-world and simulated ground truth, we verified that our reconstruction method is resilient to different agitation patterns.

Importance sampling techniques significantly reduce variance in physically based rendering. In this article, we propose a novel online framework to learn the spatial-varying distribution of the full product of the rendering equation, with a single small neural network using stochastic ray samples. The learned distributions can be used to efficiently sample the full product of incident light. To accomplish this, we introduce a novel closed-form density model, called the Normalized Anisotropic Spherical Gaussian mixture, that can model a complex light field with a small number of parameters and that can be directly sampled. Our framework progressively renders and learns the distribution, without requiring any warm-up phases. With the compact and expressive representation of our density model, our framework can be implemented entirely on the GPU, allowing it to produce high-quality images with limited computational resources. The results show that our framework outperforms existing neural path guiding approaches and achieves comparable or even better performance than state-of-the-art online statistical path guiding techniques.

We propose IM3D+, a novel approach to reconstructing 3D motion data from a flexible magnetic flux sensor array using deep learning and a structure-aware temporal bilateral filter. Computing the 3D configuration of markers (inductor-capacitor (LC) coils) from flux sensor data is difficult because the existing numerical approaches suffer from system noise, dead angles, the need for initialization, and limitations in the sensor array's layout. We solve these issues with deep neural networks to learn the regression from the simulation flux values to the LC coils' 3D configuration, which can be applied to the actual LC coils at any location and orientation within the capture volume. To cope with the influence of system noise and the dead-angle limitation caused by the characteristics of the hardware and sensing principle, we propose a structure-aware temporal bilateral filter for reconstructing motion sequences. Our method can track various movements, including fingers that manipulate objects, beetles that move inside a vivarium with leaves and soil, and the flow of opaque fluid. Since no power supply is needed for the lightweight wireless markers, our method can robustly track movements for a very long time, making it suitable for various types of observations whose tracking is difficult with existing motion-tracking systems. Furthermore, the flexibility of the flux sensor layout allows users to reconfigure it based on their own applications, thus making our approach suitable for a variety of virtual reality applications.

Motion tracking systems with viewpoint concerns or whose marker data include unreliable states have proven difficult to use despite many impactful benefits. We propose a technique inspired by active vision and using a customized hill-climbing approach to control a robot-sensor setup and apply it to a magnetic induction system capable of occlusion-free motion tracking. Our solution reduces the impact of displacement and orientation issues for markers which inherently present a dead-angle range that disturbs usability and accuracy. The resulting interface is successful in stabilizing previously unexploitable data while preventing sub-optimal states for up to hundreds of occurrences per recording and featuring an approximate 40% decrease in tracking error.

This paper proposes an interactive rear-projection tabletop glasses-free 3D display using a novel wireless magnetic motion capture system. Our tracking system employs an electromagnetic field generator and 16 magnetic detectors. It detects the 3D positions of several small markers in the generated electromagnetic field. The detectors are arranged in a ring around the rim of the conical screen of the 3D display to avoid occluding the reproduced 360-degree-viewable 3D images. For the proposed configuration, our experimental results reveal that a toroidal area around a hemispherical 3D image display area allows the 3D position to be measured with sufficient accuracy. We implemented an application to demonstrate real-time interaction with virtual 3D objects displayed on the table using markers attached to a physical object like a stick or finger.

Many motion tracking systems require solving inverse problem to compute the tracking result from original sensor measurements. For real-time motion tracking, such typical solutions as the Gauss-Newton method for solving their inverse problems need an initial value to optimize the cost function through iterations. A powerful initializer is crucial to generate a proper initial value for every time instance and, for achieving continuous accurate tracking without errors and rapid tracking recovery even when it is temporally interrupted. An improper initial value easily causes optimization divergence, and cannot always lead to reasonable solutions. Therefore, we propose a new initializer based on random-forest to obtain proper initial values for efficient real-time inverse problem computation. Our method trains a random-forest model with varied massive inputs and corresponding outputs and uses it as an initializer for runtime optimization. As an instance, we apply our initializer to IM3D[1], which is a real-time magnetic 3D motion tracking system with multiple tiny, identifiable, wireless, occlusion-free passive markers (LC coils).

We describe our approach that derives reliable 6-DOF information including the translation and the rotation of a rigid marker in a 3D space from a set of insufficient 5-DOF measurements. As a practical example, we carefully constructed a prototype and its design and evaluated it in our 3D dexterous motion-tracking system, IM6D, which is our novel real-time magnetic 3D motion-tracking system that uses multiple identifiable, tiny, lightweight, wireless, and occlusion-free markers. The system contains two key technologies; a 6-DOF computation algorithm and a marker design for 6D marker. The 6-DOF computation algorithm computes the result of complete 6-DOF information including translation and rotation in 3D space for a single rigid marker that consists of three LC coils. We propose several possible approaches for implementation, including geometric, matrix-based kinematics, and computational approaches. In addition, we introduce workflow to find an optimal marker design for the system to achieve the best compromise between its smallness and accuracy based on the tracking principle. We experimentally compare the performances of some typical marker prototypes with different layouts of LC coils. Finally, we also show another experimental result to prove the effectiveness of the results from the solutions in these two problems.

We propose Coupled-clay, a bi-directional 3D collaborative interactive environment that supports the 3D modeling work between groups of users at remote locations. Coupled-clay consists of two network-connected workspaces, the Physical Interaction Space and the Virtual Interaction Space. The physical interaction space allows a user to directly manipulate a physical object whose shape and position are precisely tracked. This tracked 3D information is transferred to the virtual interaction space in real time. The virtual interaction space is made of an interactive multi-user stereoscopic 3D tabletop, or other 3D displays with adequate interaction device. The users at the virtual interaction space observe the virtual 3D object which corresponds to the physical object and manipulate its geometrical attributes (e.g., translation, rotation and scaling). Additionally, they can control the graphical attributes of the virtual object such as color and texture. Information about changes in geometrical and graphical attributes are sent back to the physical interaction space in real time and reflected to the object in the physical interaction space by a robotic arm and a top-mounted projector. Coupled-clay can be used to remotely collaborate on 3D modeling tasks such as between a skilled designer and novice learners. This paper details our Coupled-clay implementation and presents its interaction capabilities.

We propose IM6D, a novel real-time magnetic motion-tracking system using multiple identifiable, tiny, lightweight, wireless and occlusion-free markers. It provides reasonable accuracy and update rates and an appropriate working space for dexterous 3D interaction. Our system follows a novel electromagnetic induction principle to externally excite wireless LC coils and uses an externally located pickup coil array to track each of the LC coils with 5-DOF. We apply this principle to design a practical motion-tracking system using multiple markers with 6-DOF and to achieve reliable tracking with reasonable speed. We also solved the principle's inherent dead-angle problem. Based on this method, we simulated the configuration of parameters for designing a system with scalability for dexterous 3D motion. We implemented an actual system and applied a parallel computation structure to increase the tracking speed. We also built some examples to show how well our system works for actual situations.