In this paper, a method for generating a map from path information described using natural language (textual path) is proposed. In recent years, robotics research mainly focus on vision-and-language navigation (VLN), a navigation task based on images and textual paths. Although VLN is expected to facilitate user instructions to robots, its current implementation requires users to explain the details of the path for each navigation session, which results in high explanation costs for users. To solve this problem, we proposed a method that creates a map as a topological map from a textual path and automatically creates a new path using this map. We believe that large language models (LLMs) can be used to understand textual path. Therefore, we propose and evaluate two methods, one for storing implicit maps in LLMs, and the other for generating explicit maps using LLMs. The implicit map is in the LLM’s memory. It is created using prompts. In the explicit map, a topological map composed of nodes and edges is constructed and the actions at each node are stored. This makes it possible to estimate the path and actions at waypoints on an undescribed path, if enough information is available. Experimental results on path instructions generated in a real environment demonstrate that generating explicit maps achieves significantly higher accuracy than storing implicit maps in the LLMs.

Recently, robot navigation methods using human instructions have been actively studied, including visual language navigation. Although language is one of the most promising forms of instruction, words often contain ambiguities. To complement this problem, we propose to use geometric instruction as a clue to the task goal. Specifically, in our proposed system, we assume that the robot receives a rough position of the target from human gesture. The robot adaptively estimates the reliability of this geometric instruction, and switches between exploration and instruction-following modes depending on the reliability value. We conducted evaluation of our method using a 3D simulation environment, and show that the task success rate and other metrics improve compared with the baseline methods.

Localization in topological maps is essential for image-based navigation using an RGB camera. Localization using only one camera can be challenging in medium-to-large-sized environments because similar-looking images are often observed repeatedly, especially in indoor environments. To overcome this issue, we propose a learning-based localization method that simultaneously utilizes the spatial consistency from topological maps and the temporal consistency from time-series images captured by a robot. Our method combines a convolutional neural network (CNN) to embed image features and a recurrent-type graph neural network to perform accurate localization. When training our model, it is difficult to obtain the ground truth (GT) pose of the robot when capturing images in real-world environments. Hence, we propose a sim2real transfer approach with semi-supervised learning that leverages simulator images with the GT pose in addition to real images. We evaluated the proposed method quantitatively and qualitatively and compared it with several state-of-the-art baselines. The proposed method outperformed the baselines in environments where the map contained similar images. Moreover, we evaluated an image-based navigation system incorporating our localization method and confirmed that navigation accuracy significantly improved in the simulator and real environments compared to the other baseline methods.

We present an unsupervised simultaneous learning framework for the task of monocular camera re-localization and depth estimation from unlabeled video sequences. Monocular camera re-localization refers to the task of estimating the absolute camera pose from an instance image in a known environment, which has been intensively studied for alternative localization in GPS-denied environments. In recent works, cam-era re-localization methods are trained via supervised learning from pairs of camera images and camera poses. In contrast to previous works, we propose a completely unsupervised learning framework for camera re-localization and depth estimation, requiring only monocular video sequences for training. In our framework, we train two networks that estimate the scene coordinates using directions and the depth map from each image which are then combined to estimate the camera pose. The networks can be trained through the minimization of loss functions based on our loop closed view synthesis. In experiments with the 7-scenes dataset, the proposed method outperformed the re-localization of the state-of-the-art visual SLAM, ORB-SLAM3. Our method also outperforms state-of-the-art monocular depth estimation in a trained environment.

Network traffic prediction based on probe vehicle data is important for traffic management and route recommendation and has been intensively studied. Previous traffic prediction methods mainly focused on recurring traffic congestion. Predicting non-recurring traffic congestion, caused by events and accidents, is significantly more important; however, it has not been intensively studied. To predict non-recurring traffic congestion using probe data, we need to estimate the current traffic conditions based onsparseobservations forlargetraffic networks to track traffic changesonline. Conventional traffic forecasting methods have not been able to solve all of these problems. To address these problems, we propose a data assimilation method using a state space neural network (SSNN) with an incorporated topology of road networks. The SSNN model can easily model network traffic and can easily estimate its states and parameters by data assimilation using Bayesian filtering. In this study, we adopted a decoupled extended Kalman filter (DEKF) based data assimilation, which is scalable and applicable to large-scale network traffic, to estimate the states and parameters online. We evaluate the proposed method using an open dataset that includes a road network comprising over 30000 road segments. The results show that our method achieves higher prediction accuracy for predicting unknown traffic congestion and is more robust against data sparsity than conventional state estimation methods.

Self-supervised learning for monocular depth estimation has been widely investigated as an alternative to the supervised learning approach. Uncertainty estimation in depth estimation is a crucial problem for applications, such as autonomous driving, in detecting unreliable depth. In this study, we propose a variational model to estimate depth uncertainty in self-supervised learning. Our approach leverages time-series images to handle the depth distribution from appearance variations in training. We introduce the Mahalanobis-Wasserstein distance between two consecutive frames to learn the uncertainty. In inference, our method estimates the uncertainty map and the depth image at each pixel from a single image. In experiments on KITTI, Make3D, and DIODE datasets, we show that our model achieves better uncertainty estimation than previous approaches as well as high accuracy of depth estimation

Humans can robustly follow a visual trajectory defined by a sequence of images (i.e. a video) regardless of substantial changes in the environment or the presence of obstacles. We aim at endowing similar visual navigation capabilities to mobile robots solely equipped with a RGB fisheye camera. We propose a novel probabilistic visual navigation system that learns to follow a sequence of images with bidirectional visual predictions conditioned on possible navigation velocities. By predicting bidirectionally (from start towards goal and vice versa) our method extends its predictive horizon enabling the robot to go around unseen large obstacles that are not visible in the video trajectory. Learning how to react to obstacles and potential risks in the visual field is achieved by imitating human teleoperators. Since the human teleoperation commands are diverse, we propose a probabilistic representation of trajectories that we can sample to find the safest path. We evaluate our navigation system quantitatively and qualitatively in multiple simulated and real environments and compare to state-of-the-art baselines. Our approach outperforms the most recent visual navigation methods with a large margin with regard to goal arrival rate, subgoal coverage rate, and success weighted by path length (SPL). Our method also generalizes to new robot embodiments never used during training.

Map matching is a procedure that estimates the route traveled by vehicles or people by using observed coordinates. It is an important preprocessing procedure for location services based on global positioning system (GPS) data obtained from probe vehicles. One recently proposed major map matching approach is the hidden Markov model (HMM)-based method. However, HMM-based approaches suffer from latency, because they rely on the availability of future GPS points. This latency limits the ability of real-time traffic sensing and location services. This paper presents a novel online map matching algorithm that uses a probabilistic route prediction model instead of future GPS points. The probabilistic route prediction model can be trained by using historical trajectory data. Our experimental results show that the accuracy of the untrained proposed model is competitive with a naïve online HMM-based method without any latency. Moreover, the results show that the trained model obtains even higher accuracy. The experimental results also show that the proposed method is faster than the online HMM.

Traffic anticipation enhances driving intelligence and strengthens the ability to take early vehicle control action, e.g., lane change and speed adjustment, in a dynamically varying traffic environment. This paper presents an efficient vehicle driving system, based on detailed anticipation of surrounding traffic, that aims at optimizing the driving performance of individual vehicles and smoothening traffic flows on multilane roads. More elaborately, under a connected vehicle environment, the system receives the states of all vehicles that exist within its communication range. Based on their predicted states in a look forward horizon, the system generates the optimal acceleration and makes lane change decision simultaneously in the model predictive control framework. A fast hierarchical optimization scheme is used in the framework for its onboard implementation. The proposed efficient driving system is applied to a fraction of traffic, and both the individual and overall traffic performances are evaluated using a microscopic traffic simulator. It is revealed that the vehicles under the proposed efficient driving system improve their fuel economy and travel efficiency, significantly. In the mixed traffic, by the influence of the vehicle with the proposed driving system, the other traditionally driven vehicles also improve their performance.

Anticipative control of vehicles is a potential approach for improving travel efficiency of individual vehicles, smoothing traffic flows on urban roads, alleviating impacts on the environment and elevating comforts of the users in various respects. This paper presents such a vehicle driving system in a model predictive control (MPC) framework to efficiently drive a vehicle on multi-lane roads. Anticipation enhances the driving intelligence and strengthens the vehicle's ability in taking advance action, e.g., lane change, speed adjustment, in a dynamically varying traffic environment. More elaborately, presuming a connected vehicle environment, the system receives the information form the surrounding vehicles and infrastructure instantly through V2X communication systems and, using dynamical models, predicts the future road-traffic states. Considering relevant constraints and a performance index, the system generates the optimal acceleration and executes lane change maneuver optimally if long term advantages are anticipated. Numerical simulation in realistic traffic flow conditions reveals that the vehicles with the proposed driving system improve their travel efficiency significantly.

This paper presents the development of a mathematical model of the human decision making. The main contributions of this paper are introduction of the logistic regression model as the mathematical model of the decision, development of the real time prediction of the decision based on the model, and proposal of some useful quantified measures to evaluate the characteristics of the human behavior. The proposed modeling and analysis strategies are applied to the driving behavior, in particular, focusing on the turn-right task in the intersections. Furthermore, two quantified measures “decision entropy” and “decision aggressiveness” are defined based on the estimated parameters in the logistic regression model. The objective evaluation made by these measures agrees well with the subjective evaluation made by the questionnaires.