In this paper, we introduce a framework for imitation learning in navigation that enables policy learning from one-shot images without a physical robot and facilitates the transfer of this policy from simulation to reality. Utilizing Neural Radiance Fields (NeRF), our approach generates a simulated environment and simultaneously models expert behavior. This removes the necessity for a physical robot during both the expert teaching phase and the agent’s learning process, allowing for the application of policies learned within the NeRF simulation to real-world robots. We validate our method by demonstrating the navigation with an actual robot using the policy learned by our approach. Moreover, we present a method for adapting to changes in the robot configuration, such as camera parameters and robot dimensions, by simulating adjustments in the robot configuration throughout the learning and assessing its generalizability.

We proposed a teleoperation method, illusory control (IC), that provides a comfortable operation experience using a seamless transition between real and pre-prepared virtual environments. Therefore, the mobile robot with IC could function solely in familiar environments. To make IC applicable in unfamiliar environments, this study proposes a novel method, instant IC, that eliminates the requirement for a pre-prepared virtual environment. The proposed robot system can instantly generate a virtual environment using actual 360° images of the robot in motion, utilizing instant neural graphics primitives and neural radiance fields. The 360° images allow the entire surrounding environment to be virtualized without requiring specific camera orientations. In addition, by optimizing the density of neural radiance fields using depth estimation results beforehand, the reconstruction accuracy at unknown poses can be guaranteed. Furthermore, we propose a depth scaling method based on the actual measurements obtained by LiDAR to increase the consistency of virtual and real environments. With this instant virtual environment, the proposed system enables teleoperation in unknown environments via the seamless transition between real and virtual environments. The experimental results exhibit consistent and smooth back-and-forth transitions between virtual and real space in mobile robot teleoperation.

Shared control, in which teleoperation and autonomous control are combined to move the robot, is expected to improve the efficiency of the user teleoperation problem. However, a problem exists whereby the user acceptance decreases owing to the conflict of intention between the teleoperation and autonomous control. In this study, we address this problem by providing an illusion to humans. We propose a teleoperation method named the “Illusory Control” that can achieve both mobility efficiency and user acceptance by implementing a cyber-physical system that controls a robot in real space through robot operations in virtual space. Illusory Control has two functions: the “Illusion of Intention,” which provides the illusion that the robot is operating according to human intention, and “Illusion of Time,” which provides the illusion of time to fill the gap by changing human behavior when the robot positions in the virtual space and real space diverge. Preliminary teleoperation experiments with subjects demonstrated that the system improves the operational efficiency and acceptance of the system compared to conventional teleoperation methods, namely direct teleoperation and shared control.