Autonomous valet parking (AVP) aims to help the human drivers navigate to the desired location in the parking lot. Currently, the AVP task is not flexible enough to perform the open-vocabulary navigation tasks such as "navigate to the exit" or "park near the elevator". The widely used map formats for AVP like vectorized maps have some limitations including limited semantics, high cost and poor human-machine interaction, restricting the flexible application of AVP in complex scenarios. To address these problems, we propose AVP Scene Graph (AVP-SG), a hierarchical visual language mapping and navigation framework for open-vocabulary AVP tasks, which enables autonomous navigation from multi-modal human instructions. Our framework consists of two parts: a bottom-up mapping module and a top-down navigation module. In the mapping pipeline, assisted by the vision-language model (VLM) and optical character recognition (OCR) model, we first extract open-vocabulary conceptual semantics from images and project them to the elements of map. Next, by the bottom-up scheme performing feature fusion layer by layer, the scene graph is built hierarchically, consisting of slot, lane, block, and garage layer. In the top-down navigation pipeline, the navigation goal can be efficiently found by an LLM-enhanced graph retrieval approach. Experiments on real-world AVP tasks prove that the self-driving vehicle can perform open-vocabulary AVP tasks successfully utilizing the AVP-SG.

Robust localization is the cornerstone of autonomous driving, especially in challenging urban environments where GPS signals suffer from multipath errors. Traditional localization approaches rely on high-definition (HD) maps, which consist of precisely annotated landmarks. However, building HD map is expensive and challenging to scale up. Given these limitations, leveraging navigation maps has emerged as a promising low-cost alternative for localization. Current approaches based on navigation maps can achieve highly accurate localization, but their complex matching strategies lead to unacceptable inference latency that fails to meet the real-time demands. To address these limitations, we introduce MapLocNet, a novel transformer-based neural re-localization method. Inspired by image registration, our approach performs a coarse-to-fine neural feature registration between navigation map features and visual bird’s-eye view features. MapLocNet substantially outperforms the current state-of-the-art methods on both nuScenes and Argoverse datasets, demonstrating significant improvements in localization accuracy and inference speed across both single-view and surround-view input settings. We highlight that our research presents an HD-map-free localization method for autonomous driving, offering a costeffective, reliable, and scalable solution for challenging urban environments.

The end-to-end neural network has become a hot topic in recent years. Compared with traditional module-based solutions, the end-to-end paradigm is able to reduce the accumulated error and avoid information loss, so that it earns great attention in autonomous driving tasks. However, the current end-to-end network designs easily lose useful information during training due to the complexity of mapping high-dimensional visual observation to navigation waypoints. Since the future navigation point is reasoned from the former one, the planning task is like a sequence generation task. Inspired by the great power of the neural language model, we propose an end-to-end framework, which transfers the planning task as a language sequence generation task conditioned on pixel inputs. The proposed method firstly extracts and transforms the image feature from camera-view to bird-eye-view (BEV). Then the target navigation point is constructed into a text sequence, as the prompt of the visual-language transformer. Finally, the auto-regressive transformer decoder receives the BEV feature and the text sequences to generate sequential waypoints. Overall, our proposed method can make full use of the environmental information and express the planning trajectory as a language sequence to learn the correspondence between trajectory sequences and images. We have conducted extensive experiments on CARLA benchmarks and our model achieves state-of-the-art performance compared with other visual methods.

Accurate localization ability is a crucial component for autonomous robots. Given existing LiDAR 3D points maps, it is cost-effective to localize the robot only with onboard camera compared to LiDAR. However, matching 2D visual information with 3D point cloud maps presents huge challenges due to different modalities, dimensions, noise and occlusion issues. To overcome it, we propose an end-to-end neural network-based solution, which determines the 6-DoF pose of the camera relative to an existing LiDAR map with centimeter accuracy. Given a query image, a pre-acquired point cloud and an initial pose, the cross-modality network will output a precise pose. By projecting the 3D point cloud onto the image plane, a depth image is acquired as seen from the initial pose. Subsequently, a cross-modality flow network establishes the correspondences of 2D pixels and projected points. Importantly, we leverage a robust probabilistic Perspective-n-Point (PnP) module, which are capable of fine-tuning 2D pairs and learning the pairs weight in an end-to-end manner. A comprehensive evaluation of our proposed algorithm is conducted in KITTI datasets. Furthermore, deploying the algorithm on the real-world parking lot scenario validates its strong practicality of the proposed algorithm. We highlight that this research offers a cost-effective and highly accurate solution that can be readily deployed in low-cost commercial vehicles.

Autonomous parking is a crucial application for intelligent vehicles, especially in crowded parking lots. The confined space requires highly precise perception, planning, and control. Currently, the traditional Automated Parking Assist (APA) system, which utilizes geometric-based perception and rule-based planning, can assist with parking tasks in simple scenarios. With noisy measurement, the handcrafted rule often lacks flexibility and robustness in various environments, which performs poorly in super crowded and narrow spaces. On the contrary, there are many experienced human drivers, who are good at parking in narrow slots without explicit modeling and planning. Inspired by this, we expect a neural network to learn how to park directly from experts without handcrafted rules. Therefore, in this paper, we present an end-to-end neural network to handle parking tasks. The inputs are the images captured by surrounding cameras and basic vehicle motion state, while the outputs are control signals, including steer angle, acceleration, and gear. The network learns how to control the vehicle by imitating experienced drivers. We conducted closed-loop experiments on the CARLA Simulator to validate the feasibility of controlling the vehicle by the proposed neural network in the parking task. The experiment demonstrated the effectiveness of our end-to-end system in achieving the average position and orientation errors of 0.3 meters and 0.9 degrees with an overall success rate of 91%. The code is available at: https://github.com/qintonguav/e2e-parking-carla

Sensing environmental obstacles and establishing an occupancy map of surroundings are critical to achieving automated parking for autonomous vehicles. This paper presents a method to obtain surrounding occupancy information from inverse perspective mapping (IPM) images. This method uses the easily-accessed pseudo-labels from LiDAR to supervise a visual network, which can detect occupied boundaries of obstacles. Fusing this visual occupancy with ego-motion information, we develop a multi-frame fusion approach to build a local OGM to realize online environment mapping. Compared with other learning-based occupancy approaches, our method does not require time-consuming and labor-intensive labeling for the environment due to the ground truth of surrounding occupancy coming from LiDAR easily. The proposed method achieves LiDAR-like performance with pure visual inputs, which greatly decreases the cost of real products. Experiments on driving and parking environments prove that our method can accurately sense surrounding occupancy information and build a robust occupancy map of the environment.