Place recognition is a crucial component that enables autonomous vehicles to obtain localization results in GPS-denied environments. In recent years, multimodal place recognition methods have gained increasing attention. They overcome the weaknesses of unimodal sensor systems by leveraging complementary information from different modalities. Most existing methods explore cross-modality correlations through feature-level or descriptor-level fusion. Conversely, the recently proposed 3D Gaussian Splatting provides a new perspective on multimodal spatio-temporal fusion by harmonizing temporally continuous multimodal data into an explicit scene representation. In this paper, we propose a 3D Gaussian Splatting-based multimodal place recognition network dubbed GSPR. It explicitly combines multi-view RGB images and LiDAR point clouds into a spatio-temporally unified scene representation with the proposed Multimodal Gaussian Splatting. A network composed of 3D graph convolution and transformer is designed to extract global descriptors from the Gaussian scenes for place recognition. Extensive evaluations on three datasets demonstrate that our method can effectively leverage complementary strengths of both multi-view cameras and LiDAR, achieving SOTA place recognition performance while maintaining solid generalization ability. Our open-source code will be released at https://github.com/QiZS-BIT/GSPR.

Maps play an important role in autonomous driving systems. The recently proposed 3D Gaussian Splatting (3D-GS) produces rendering-quality explicit scene reconstruction results, demonstrating the potential for map construction in autonomous driving scenarios. However, because of the time and computational costs involved in generating Gaussian scenes, how to update the map becomes a significant challenge. In this paper, we propose LT-Gaussian, a map update method for 3D-GS-based maps. LT-Gaussian consists of three main components: Multimodal Gaussian Splatting, Structural Change Detection Module, and Gaussian-Map Update Module. Firstly, the Gaussian map of the old scene is generated using our proposed Multimodal Gaussian Splatting. Subsequently, during the map update process, we compare the outdated Gaussian map with the current LiDAR data stream to identify structural changes. Finally, we perform targeted updates to the Gaussian-map to generate an up-to-date map. We establish a benchmark for map updating on the nuScenes dataset to quantitatively evaluate our method. The experimental results show that LT-Gaussian can effectively and efficiently update the Gaussian-map, handling common environmental changes in autonomous driving scenarios. Furthermore, by taking full advantage of information from both new and old scenes, LT-Gaussian is able to produce higher quality reconstruction results compared to map update strategies that reconstruct maps from scratch. Our open-source code is available at https://github.com/ChengLuqi/LT-gaussian.

LiDAR-based place recognition (LPR) is one of the most crucial components of autonomous vehicles to identify previously visited places in GPS-denied environments. Most existing LPR methods use mundane representations of the input point cloud without considering different views, which may not fully exploit the information from LiDAR sensors. In this article, we propose across-viewtransformer-based network, dubbed CVTNet, to fuse the range image views and bird's eye views generated from the LiDAR data. It extracts correlations within the views using intratransformers and between the two different views using intertransformers. Based on that, our proposed CVTNet generates a yaw-angle-invariant global descriptor for each laser scan end-to-end online and retrieves previously seen places by descriptor matching between the current query scan and the prebuilt database. We evaluate our approach on three datasets collected with different sensor setups and environmental conditions. The experimental results show that our method outperforms the state-of-the-art LPR methods with strong robustness to viewpoint changes and long-time spans. Furthermore, our approach has better real-time performance that can run faster than the typical LiDAR frame rate does.

In this paper, we introduce a practical two-layer planning framework for autonomous vehicles operating in unknown off-road environment. The first layer refers to a global path planning layer, searching a shortest global path from road network according to given task points. We build up road network through pre-processing on Google earth and refactoring network in terms of vehicle real historical trajectory on real-time. The second layer refers to a local planning layer, solving a real-time planning problem to generate a collision-free and kinematic-feasible local path by a hybrid trajectory planning method. Our method has been verified in real off-road environment. Experimental results show that the proposed planning method performs well in off-road environment.

This paper presents an Enhanced Adaptive Cruise Control (EACC) framework that can work in different modes according to the forward targets. The EACC system, which was proposed in this paper, is based on a unified model and can achieve speed tracking, stop & go and autonomous emergency braking (AEB). Notably, speed tracking does not require a real preceding vehicle, a virtual vehicle can be set in front of the EACC vehicle. The mathematical method of setting the virtual preceding vehicle and the switching logic between the different working modes of the EACC system were given. Employing a constraints softening method to avoid computing infeasibility, an optimal control law is numerically calculated using the CVXGEN solver. Finally, real vehicle tests show that the EACC framework provides significant benefits in terms of speed-tracking capability, safety and comfort requirements while satisfying driver desired car following characteristics for different driving situations.

Simultaneous localization and mapping (SLAM), as an important tool for vehicle positioning and mapping, plays an important role in the unmanned vehicle technology. This paper mainly presents a new solution to the LIDAR-based SLAM for unmanned vehicles in the off-road environment. Many methods have been proposed to solve the SLAM problems well. However, in complex environment, especially off-road environment, it is difficult to obtain stable positioning results due to the rough road and scene diversity. We propose a SLAM algorithm based on grid which combining probability and feature by Expectation-maximization (EM). The algorithm is mainly divided into three steps: data preprocessing, pose estimation, updating feature grid map. Our algorithm has strong robustness and real-time performance. We have tested our algorithm with our datasets of the multiple off-road scenes which obtained by LIDAR. Our algorithm performs pose estimation and feature map updating in parallel, which guarantees the real-time performance of the algorithm. The average processing time of each frame is about 55ms, and the average relative translation error is around 0.94%. Compared with several state-of-the-art algorithms, our algorithm has better performance in robustness and location accuracy.

Simultaneous Localization And Mapping (SLAM) plays a more and more important role in the environment perception system of Unmanned Ground Vehicle (UGV), most SLAM technologies used to be applied indoor or in urban scenarios, we present a real-time 6D SLAM approach suitable for large scale natural terrain with the help of an Inertial Measurement Unit(IMU) and two 3D Lidars. Besides dividing the entire map into many submaps which consists of large numbers of tree structure based voxels, we use probabilistic methods to represent the possibility of one voxel being occupied/null. A Sparse Pose Adjustment (SPA) method has been used to solve 6D global pose optimization with some relative poses as pose constraints and relative motions computed from IMU data as kinetics constraints. A place recognition method integrated a method named Rotation Histogram Matching (RHM) and a Branch and Bound Search (BBS) based Iterative Closest Points (ICP) algorithm is applied to realize a real-time loop closure detection. We complete global pose optimization with the help of Ceres. Experimental results obtained from a real large scale natural environment shows an effective reduction for Lidar odometry pose accumulative error and a good performance for 3D mapping.

This paper proposes the decision-making framework of lane change behavior based on Hierarchical State Machine (HSM) and we build distributed control system architecture based on RCS (Real-Time Control System) to test the model. Environment perception module, decision planning module and execution control module are put into the distributed system architecture based on RCS to improve real-time and ensure that several modules run simultaneously. Besides, the decision-making framework of lane change behavior consists of two parts: miniature scene information model and decision-making model of lane change behavior based on multi-attribute decision-making. The decision-making model of lane change behavior is based on HSM and it sets two top-level state machines: free lane change model and mandatory lane change model. Free lane change model changes the state by using lane reward model to judge and assess driving condition of each lane, while mandatory lane change model uses strategy of multi-source information fusion tojudge whether to lane change. In the end, the unmanned platform BYD Tang using vehicle embedded platform is used to verify the reliability and effectiveness of the lane change decision-making algorithm proposed in this paper in the real road environment.

In this paper, an optimization method of gear shift schedule for electric buses equipped with 4-AMT is proposed based on Dynamic Programming (DP) to improve the energy economy of the vehicle. A gear shift schedule that can be used in real-vehicle is extracted based on analysis of the obtained optimal gear shift points by DP approach in Chinese typical urban driving-cycles. Compared to the traditional two-parameter gear shift schedule in both simulation and real vehicle platform, the extracted gear shift schedule is proved to improve the energy economy of the electric vehicles (EVs) obviously.

In this paper, we propose a novel, efficient stereo visual-odometry algorithm for ground vehicles moving in outdoor environments. To avoid the drawbacks of computationally-expensive outlier-removal steps based on random-sample schemes, we use a single-degree-of-freedom kinematic model of the vehicle to initialize an Iterative Closest Point (ICP) algorithm that is utilized to select high-quality inliers. The motion is then computed incrementally from the inliers using a standard linear 3D-to-2D pose-estimation method without any additional batch optimization. The performance of the approach is evaluated against state-of-the-art methods on both synthetic data and publicly-available datasets (e.g., KITTI and Devon Island) collected over several kilometers in both urban environments and challenging off-road terrains. Experiments show that the our algorithm outperforms state-of-the-art approaches in accuracy, runtime, and ease of implementation.

Complex robotic systems often have to operate in large environments. At the same time, their dynamic is complex enough that path planning algorithms need to reason about the kinodynamic constraints of these systems. On the other hand, such robotic systems are typically expected to operate with speed that is commensurate with that of humans. This poses stringent limitation on available planning time. These will result in a contradiction between planning efficiency and the dimensions of the state space determined by the kinodynamic constraints. In this paper we present an anytime path planning algorithm to solving this problem. First, a graduated state space which consists of state lattices and grids is constructed for planning. Then, ARA* algorithm is utilized to search the graduated state space to find a path that satisfies the kinodynamic constraints and available runtime of planning.

Two grading criteria used in determining shrimp product quality and value by the shrimp industry are: 1. Presence or percentage of black spot, measured as a percentage of the total body surface. 2. Shape quality referring to whole shrimp and broken pieces. Black spots (melanoma) on the shrimp surface are evidence of aging shrimp and are considered defects that must be removed from the main production line. Shape quality is measured as the size and the completeness of the body. Broken shrimp pieces are considered a product defect and also must be removed from the main production line. Black spot detection is a simple task for a well-designed machine vision system, which provides consistent and controlled lighting. Shape analysis, on the other hand, is a challenging task because it involves contour extraction and shape analysis. In this paper, a simple, fast, and accurate shape analysis method using Turn Angle Cross-correlation is developed for shrimp quality evaluation. Our analysis results validate that the performance of the proposed shape analysis method is suitable for real-time inspection for commercial applications.

A smart phone camera based indoor guidance system to aid the visually impaired is presented. Most proposed systems for aiding the visually impaired with indoor navigation are not feasible for widespread use due to cost, usability, or portability. A smart phone vision- based indoor guidance system that is simple, accessible, inexpensive, and discrete is developed to aid the visually impaired to navigate unfamiliar environments such as public buildings. The system consists of a smart phone and a server. The smart phone captures and transmits pictures of the user's surroundings to the server. The server processes the images and matches them to a database of stored images of the building. After matching features, the location and orientation of the person is calculated using 3D location correspondence data stored for features of each image. Positional information is then transmitted back to the smart phone and communicated to the user via text-to-speech. This paper focuses on developing the vision technology for this unique application rather than building the complete system. Experimental results demonstrate the ability of the system to quickly and accurately determine the pose of the user in a university building.

This paper presents a novel method to realize on-board visual odometry system. Vehicular kinematic constrain is used in the motion estimation algorithms. The work is a extension from planar steering model to 3-dof in which vehicle's motion is modeled more reasonable and accurate. By virtue of appropriate simplification, the close-form solution of motion parameters can be obtained only need to find real roots of a cubic equation. Then optimization based refine method can bring the winner solution to accurate solution utilizing inliers founded. The algorithm has been tested on both simulation platform and real car test and achieved promising results.

The 2011 Intelligent Vehicle Future Challenge (11'IVFC) tested self-driving systems in real urban scenarios. The entry of Beijing Institute of Technology: BIT-III finished the 10-kilometer long track in 28 minutes without human operation and obeyed traffic regulations in most circumstances. This paper presented the design and implementation of BIT-III. As a universal self-driving system, BIT-III valued extensibility and featured modularized system architecture. For a better compatibility with diverse sensing devices, BIT-III classified perception to be either OGM (Occupancy Grid Map)-oriented or object-oriented based on the output mode. To work in environments with uncertainties, BIT-III gave first priority to safety and stability in driving, and realized them in the core-level components as the instinct of the system. Even in the unknown environment in the ll'IVFC, BIT-III was able to drive smoothly without crashes.

We present an iterative linear quadratic regulator (ILQR) method for trajectory tracking control of a wheeled mobile robot system. The proposed scheme involves a kinematic model linearization technique, a global trajectory generation algorithm, and trajectory tracking controller design. A lattice planner, which searches over a 3D ( x, y, θ ) configuration space, is adopted to generate the global trajectory. The ILQR method is used to design a local trajectory tracking controller. The effectiveness of the proposed method is demonstrated in simulation and experiment with a significantly asymmetric differential drive robot. The performance of the local controller is analyzed and compared with that of the existing linear quadratic regulator (LQR) method. According to the experiments, the new controller improves the control sequences ( ν, ω ) iteratively and produces slightly better results. Specifically, two trajectories, ‘S’ and ‘8’ courses, are followed with sufficient accuracy using the proposed controller.

This paper presents an approach to detect and recognize traffic signs present in the urban scenes in China. The algorithm is composed of three steps that are color segmentation, shape detection and pictogram recognition. In the first step Ridge Regression is used to obtain a precise segmentation in RGB color space and achieves the same good performance as many machine learning based methods while using less computation time. Recognition process include a novel feature extraction involves OTSU method, and the feature extracted is robust against illumination variations and distortions. The algorithm has been run on several thousands of images with promising results.

Color grading is a crucial step in the processing of fruits and vegetables that directly affects profitability, because the quality of agricultural products is often associated with their color. Most existing automatic color grading systems determine color quality either by directly comparing product color against a predefined and fixed set of reference colors or by using a set of color separating parameters, often in three-dimensional color spaces. Using these methods, it is not convenient for the user to adjust color preferences or grading parameters. In this paper, we present an effective and user-friendly color mapping concept for automated color grading that is well suited for commercial production. User friendliness is often viewed by the industry as a very important factor to the acceptance and success of automation equipment. This color mapping method uses preselected colors of interest specific to a given application to calculate a unique set of coefficients for color space conversion. The three-dimensional RGB color space is converted into a small set of color indices unique to the application. In contrast with more complex color grading techniques, the proposed method makes it easy for a human operator to specify and adjust color-preference settings Tomato and date maturity evaluation and date surface defect detection are used to demonstrate the performance of this novel color mapping concept.

Many computer vision algorithms for pattern matching, object tracking, and 3-D reconstruction, etc., begin with feature detection and matching. Common feature detectors such as Harris, Sobel, Canny, and Difference of Gaussians perform basic linear algebra operations on an image in order to identify "corners" or "edges" for matching. These detectors however, require single-channel (grayscale) source images. For color source images, color channels are typically averaged or converted to create grayscale images for processing, discarding a large amount of highly useful information. This paper outlines the proposed color Difference of Gaussians (DoG) algorithm for feature detection, which outperforms the grayscale DoG in number and quality of features found. The new color DoG is applied to a color feature matching application for improving the library inventory (shelf-reading) process. Experimental results demonstrate the robustness of this color feature detector.

Rank and census transforms provide high resistance to radiometric distortion, vignette, and noise because they are based on the relative ordering of local pixel intensity values rather than the pixel values themselves. These transforms are widely used in many computer vision applications. An important step of computing these transforms is to compare or rank two grayscale values, which is very much like measuring color difference in color image. Color difference between two color points at any part of a uniform color space corresponds to the perceptual difference between the two colors by the human vision system. Based on this idea, we propose to use perceptual color contrast to implement color rank and census transforms and achieve this without significantly increasing the amount of data to process and without complicated computations. Furthermore, we demonstrate the feasibility of using these new transforms to find correspondences for stereo vision.

The recognition and tracking of traffic lights for intelligent vehicles based on a vehicle-mounted camera are studied in this paper. The candidate region of the traffic light is extracted using the threshold segmentation method and the morphological operation. Then, the recognition algorithm of the traffic light based on machine learning is employed. To avoid false negatives and tracking loss, the target tracking algorithm CAMSHIFT (Continuously Adaptive Mean Shift), which uses the color histogram as the target model, is adopted. In addition to traffic signal pre-processing and the recognition method of learning, the initialization problem of the search window of CAMSHIFT algorithm is resolved. Moreover, the window setting method is used to shorten the processing time of the global HSV color space conversion. The real vehicle experiments validate the performance of the presented approach.

The first Chinese unmanned ground vehicles competition - The 2009 Future Challenge: Intelligent Vehicles and Beyond (FC'09) pushed China's unmanned vehicles out of laboratories and into application environments. In order to further promote the development of unmanned vehicle technologies, the test and evaluation system for unmanned vehicles needs to be studied. The design method of test environment is proposed in accordance with the definition and classification of test environment elements. Based on the multi-platform and multi-sensor, an omnidirectional video monitoring test system of unmanned vehicles is built. The fuzzy comprehensive evaluation method combined with AHP (analytic hierarchy process) is applied to the comprehensive evaluation of unmanned vehicles. The evaluation examples of unmanned vehicles show that the proposed evaluation system can quantitatively evaluate the overall technical performance and individual technical performance of unmanned vehicles.

The 2009 Future Challenge - Intelligent Vehicle and Beyond (FC'09) was held in Xi'an, China. Our intelligent vehicle named BIT participated in all competitions at this event. This paper describes BIT's system structure and its capabilities. BIT combines a global path planning method and local path planning to drive the vehicle to address the challenges posted by the unknown competition environment. A novel curve tracking strategy based on preview and curve bisector is developed for complex paths such as U-turn. For recognizing traffic lights, Haar feature and AdaBoost algorithm are used to train and obtain traffic light classifiers. Normalization of every candidate region in RGB and HSV spaces is performed and compared with a threshold to fulfill the verification. The experiment describes BIT's performance and the conclusion sets forth the main work in the next step.

In this paper, a 3D laser point cloud-based navigation method for autonomous ground vehicles in complex environment is proposed. With a coordinate transformation of laser data from sphere to cylinder, environment perception cylinder is configured. In the cylinder, terrain traversability is predicted through analysis on radial and tangential slope of 3D point cloud. In addition, the candidate point cloud of traversable region has been extracted. Furthermore, navigation circle contained with direction information is built up based on the candidate point cloud. Extended experimental results demonstrate that the method allows autonomous ground vehicle to move safely and correctly in complex environment.

Road detection is an important problem with application to driver assistance systems and autonomous, self-guided vehicles. The focus of this paper is on the problem of feature extraction and classification for front-view road detection. Specifically, we propose using Support Vector Machines (SVM) for road detection and effective approach for self-supervised online learning. The proposed road detection algorithm is capable of automatically updating the training data for online training which reduces the possibility of misclassifying road and non-road classes and improves the adaptability of the road detection algorithm. The algorithm presented here can also be seen as a novel framework for self-supervised online learning in the application of classification-based road detection algorithm on intelligent vehicle.

Many people die each year in the world in single vehicle roadway departure crashes caused by driver inattention, especially on the freeway. Lane Departure Warning System (LDWS) is a useful system to avoid those accident, in which, the lane detection is a key issue. In this paper, after a brief overview of existing methods, we present a robust lane detection algorithm based on geometrical model and Gabor filter. This algorithm is based on two assumptions: the road in front of vehicle is approximately planar and marked which are often correct on the highway and freeway where most lane departure accidents happen. The lane geometrical model we build in this paper contains four parameters which are starting position, lane original orientation, lane width and lane curvature. The algorithm is composed of three stages: the first stage is called off-line calibration which just runs once after the camera is mounted and fixed in the vehicle. The parameters of camera used for lane detection is accurately estimated by the 2D calibration method; The second stage is called lane model parameters estimation and lane model candidates construction, the first three parameters, starting position, lane original orientation and lane width will be estimated using dominant orientation estimation and local Hough transform. Then the construction of lane model candidates is implemented for the final lane model matching; the third stage is model matching. The proposed lane module matching algorithm is implemented to match the best fitted lane model. The combination of these modules can overcome the universal lane detection problems due to inaccuracies in edge detection such as shadow of tree and passengers on the road. Experimental results on real road will be presented to prove the effectiveness of the proposed lane detection algorithm.