In this paper, we address a method that integrates reinforcement learning into the Monte Carlo tree search to boost online path planning under fully observable environments for automated parking tasks. Sampling-based planning methods under high-dimensional space can be computationally expensive and time-consuming. State evaluation methods are useful by leveraging the prior knowledge into the search steps, making the process faster in a real-time system. Given the fact that automated parking tasks are often executed under complex environments, a solid but lightweight heuristic guidance is challenging to compose in a traditional analytical way. To overcome this limitation, we propose a reinforcement learning pipeline with a Monte Carlo tree search under the path planning framework. By iteratively learning the value of a state and the best action among samples from its previous cycle’s outcomes, we are able to model a value estimator and a policy generator for given states. By doing that, we build up a balancing mechanism between exploration and exploitation, speeding up the path planning process while maintaining its quality without using human expert driver data.

Intra-driver and inter-driver heterogeneity has been confirmed to exist in human driving behaviors by many studies. This research proposes a driver identification method by modeling such heterogeneities in car following sequences. It is assumed that all drivers share a pool of driver states; under each state, a car-following data sequence obeys a specific probability distribution in feature space; each driver has his/her own probability distribution over the states, called driver profile, which characterize the intra-driver heterogeneity, while the difference between the driver profile of different drivers depicts the inter-driver heterogeneity. Thus, the driver profile can be used to distinguish a driver from others. Based on the assumption, a method of driver identification is proposed to take both intra- and inter-driver heterogeneity into consideration, and a method is developed to jointly learn parameters in behavioral feature extractor, driver states, and driver profiles. Experiments demonstrate the performance of the proposed method in driver identification on naturalistic car-following data: accuracy of 82.3% is achieved in an 8-driver experiment using 10 car-following sequences of duration 15 seconds for online inference. The potential of fast registration of new drivers is demonstrated and discussed.

Localization is an essential technique in mobile robotics. In a complex environment, it is necessary to fuse different localization modules to obtain more robust results, in which the error model plays a paramount role. However, exteroceptive sensor-based odometries(ESOs), such as LiDAR/visual odometry, often deliver results with scene-related error, which is difficult to model accurately. To address this problem, this research designs a scene-aware error model for ESO, based on which a multimodal localization fusion framework is developed. In addition, an end-to-end learning method is proposed to train this error model using sparse global poses such as results from global positioning system(GPS) and inertial measurement unit(IMU). The proposed method is realized for error modeling of LiDAR/visual odometry, and the results are fused with dead reckoning to examine the performance of vehicle localization. Experiments are conducted using both simulation and real-world data of experienced and unexperienced environments, and the experimental results demonstrate that with the learned scene-aware error models, vehicle localization accuracy can be largely improved and shows adaptiveness in unexperienced scenes.

Driving in a human-like manner is important for an autonomous vehicle to be a smart and predictable traffic participant. To achieve this goal, parameters of the motion planning module should be carefully tuned, which needs great effort and expert knowledge. In this study, a method of learning cost parameters of a motion planner from naturalistic driving data is proposed. The learning is achieved by encouraging the selected trajectory to approximate the human driving trajectory under the same traffic situation. The employed motion planner follows a widely accepted methodology that first samples candidate trajectories in the trajectory space, then select the one with minimal cost as the planned trajectory. Moreover, in addition to traditional factors such as comfort, efficiency and safety, the cost function is proposed to incorporate incentive of behavior decision like a human driver, so that both lane change decision and motion planning are coupled into one framework. Two types of lane incentive cost — heuristic and learning based — are proposed and implemented. To verify the validity of the proposed method, a data set is developed by using the naturalistic trajectory data of human drivers collected on the motorways in Beijing, containing samples of lane changes to the left and right lanes, and car followings. Experiments are conducted with respect to both lane change decision and motion planning, and promising results are achieved.

This work addresses on the following problem: given a set of unsynchronized history observations of two scenes that are correlative on their dynamic changes, the purpose is to learn a cross-scene predictor, so that with the observation of one scene, a robot can onlinely predict the dynamic state of the other. A method is proposed to solve the problem via modeling dynamic correlation using latent space shared auto-encoders. Assuming that the inherent correlation of scene dynamics can be represented by shared latent space, where a common latent state is reached if the observations of both scenes are at an approximate time. A learning model is developed by connecting two auto-encoders through the latent space, and a prediction model is built by concatenating the encoder of the input scene with the decoder of the target one. Simulation datasets are generated imitating the dynamic flows at two adjacent gates of a campus, where the dynamic changes are triggered by a common working and teaching schedule. Similar scenarios can also be found at successive intersections on a single road, gates of a subway station, etc. Accuracy of cross-scene prediction is examined at various conditions of scene correlation and pairwise observations. Potentials of the proposed method are demonstrated by comparing with conventional end-to-end methods and linear predictions.

Terrain traversability analysis is a fundamental issue to achieve the autonomy of a robot at off-road environments. Geometry-based and appearance-based methods have been studied in decades, while behavior-based methods exploiting learning from demonstration (LfD) are new trends. Behavior-based methods learn cost functions that guide trajectory planning in compliance with experts' demonstrations, which can be more scalable to various scenes and driving behaviors. This research proposes a method of off-road traversability analysis and trajectory planning using Deep Maximum Entropy Inverse Reinforcement Learning. To incorporate the vehicle's kinematics while solving the problem of exponential increase of state-space complexity, two convolutional neural networks, i.e., RL ConvNet and Svf ConvNet, are developed to encode kinematics into convolution kernels and achieve efficient forward reinforcement learning. We conduct experiments in off-road environments. Scene maps are generated using 3D LiDAR data, and expert demonstrations are either the vehicle's real driving trajectories at the scene or synthesized ones to represent specific behaviors such as crossing negative obstacles. Different cost functions of traversability analysis are learned and tested at various scenes of capability in guiding the trajectory planning of different behaviors. We also demonstrate the peformance and computation efficiency of the proposed method.

This work studies the semantic segmentation of 3D LiDAR data in dynamic scenes for autonomous driving applications. A system of semantic segmentation using 3D LiDAR data, including range image segmentation, sample generation, inter-frame data association, track-level annotation, and semi-supervised learning, is developed. To reduce the considerable requirement of fine annotations, a CNN-based classifier is trained by considering both supervised samples with manually labeled object classes and pairwise constraints, where a data sample is composed of a segment as the foreground and neighborhood points as the background. A special loss function is designed to account for both annotations and constraints, where the constraint data are encouraged to be assigned to the same semantic class. A dataset containing 1838 frames of LiDAR data, 39 934 pairwise constraints and 57 927 human annotations is developed. The performance of the method is examined extensively. The qualitative and quantitative experiments show that the combination of a few annotations and large amount of constraint data significantly enhances the effectiveness and scene adaptability, resulting in greater than 10% improvement.

Localization is an essential technique for many robotic tasks such as mapping and navigation. Scan matching has been fused with other sensors to solve the problem at GPS restricted areas, where an accurate error model describing matching precision at various scenes is indispensable. We proposed an end-to-end method to learn a scene adaptive error model of LiDAR scan matching. A CNN (Convolutional Neural Network) is learnt to map from a LiDAR scan to an information matrix of the matching result, and a localization framework is proposed to fuse the results of LiDAR scan matching based on its error model. Experiments are conducted using both simulated and real world data, where the former is to validate the proposed method of its adaptability at various simple but typical scenes, while the later is to examine the method's practicability at real world environments. We demonstrate the performance of learning covariance error model, and examine the localization accuracy by comparing with other traditional methods. Efficiency of the proposed method is demonstrated.

Heterogeneity exists in car following behaviors due to the driver's habit, fatigue, distraction, or surrounding traffic. This research proposes a method of modeling and reasoning heterogeneous car-following behaviors based on a stochastic system, where scene vehicles are involved explicitly in addition to the traditional leader-follower pair in characterizing driving situations, a hidden variable (driver state) is introduced to conjugate driving situations to heterogeneous models in predicting a driver's acceleration control, and the dynamic procedure is described using a dynamic Bayesian network. Experiments are conducted using a large set of naturalistic driving data that were collected by driving an instrumented vehicle on the multi-lane motorways in Beijing, where four distinctive driver states are learnt from data, characterizing the car-following procedure with normal, slow responsive, strong and prompt responsive, and unresponsive behavioral styles. By using the proposed scene-aware multi-state model for acceleration prediction, the error is reduced to 0.19 m/s2in average compared with 0.29 m/s2of a single-state model. Influence of scene vehicles on a driver state and subsequently on velocity control is verified based on the data. To the best of our knowledge, this is the first work that explicitly incorporates scene vehicles as influential factors in a probabilistic approach for modeling and reasoning heterogeneous car-following behaviors, and the performance is demonstrated on a large set of naturalistic driving data.

Trajectory planning has generally been framed as finding the lowest cost one from a set of trajectory candidates, where the cost function has been hand-crafted with carefully tuned parameters by experts. Such methods have technological feasibility of achieving vehicle autonomy, while the resultant behaviors could be much different with those of human drivers. This research proposes a humanlike trajectory planning method by learning from naturalistic driving data. A cost function is formulated by incorporating not only the components on comfort, efficiency and safety, but also lane incentive by referring to a human driver's lane change decisions. Coefficients of the cost components are learnt by correlating the probability of a trajectory being selected with its distance (i.e. similarity) to the human driven one at the same driving situation. A data set is developed by using the naturalistic data of human drivers on the motorways in Beijing, containing samples of lane changes to the left and right lanes, and car followings. Experiments are conducted on three aspects: 1) lane change trajectory planning to a given target lane; 2) lane change trajectory planning with simultaneous decision of a target lane; and 3) trajectory planning with simultaneous decision of maneuver. Promising results are presented.

Human-like driving is of great significance for safety and comfort of autonomous vehicles, but existing trajectory planning methods for on-road vehicles rarely take the similarity with human behavior into consideration. From a representative trajectory-generation-based planning algorithm, this paper analyzes the systematic deviation of the generated trajectories from human trajectories, and proposes a new scheme of trajectory generation by compensating the deviation using a deviation profile learned from data. Experimental results show that the proposed trajectory generator is able to fit the human trajectories considerably better than the original one with only one additional degree of freedom. When used for online trajectory planning, with the same level of computational complexity, the proposed generator is able to generate trajectories that are more human-like than original generator does, which provides basis for autonomous vehicle to perform human-like trajectory planning.

This study proposes a multi-level trajectory analysis method for modeling traffic behavior from an ego-centric view, where on-road vehicle trajectories are collected based on the authors' previous studies of an on-board system consisting of multiple 2D lidar sensors. From an input set of trajectories, a set of hot regions (topics) that trajectory points most frequently present are first discovered using a sticky HDP-HMM; then, the major paths of the trajectories' transitions across different hot regions are extracted by recursively mining frequent subsequences of topics; and finally, paths are modeled using a hierarchical hidden Markov model (HHMM), where the intra-path dynamics is represented using an HMM, in which each state corresponds to a hot region, while the inter-path transition is assumed to be Markovian. The model could be used for behavior prediction, i.e. whenever a vehicle is detected in a scene, predicting which route it will probably follow and how its trajectory will probably develop over time, which is essential to interpreting the potential risks for longer time horizons. Experiments are conducted using a large set of vehicle trajectories collected from motorways in Beijing, and promising results are presented.

This research represents the heterogeneity in car following by a hidden variable driver state, which could change due to the driver's habit, fatigue, distraction, influence of surrounding traffic etc, resulting in the heterogeneous behaviors of such as fast or slow, strong or weak response to the same level of stimuli. A probabilistic method of driver state understanding is proposed by modeling and reasoning the heterogeneity in car-following behaviors, and the influence of surrounding traffic is addressed explicitly in addition to the leader-follower pair aiming at applications in crowded real-world traffic. Experiments are conducted by using the on-road trajectory data that were collected from motorways in Beijing, where four distinctive driver states and corresponding car-following models are learnt. With online understanding of driver state, the particular car-following model is used to predict the drivers velocity control, where results of improved accuracy are demonstrated.

This two-part paper aims to study lane change behaviors at the tactical level from an on-road perspective. Compared with longitudinal driving tasks, a lane change is more complicated because this task has more interactions with surrounding vehicles; thus, there are more potential risks during this procedure. Based on the results from Part I on an on-road vehicle trajectory collection, this part investigates lane change extraction and scene-based behavior analysis, and it has a particular focus on understanding the interactions between an ego and surrounding vehicles during the procedure. We claim that this paper provides the following novel contributions: 1) an automatic method is proposed for extracting lane change segments from a continuous driving sequence by modeling and recognizing patterns in a steering angle; 2) a lane change database at the trajectory level is generated, which reflects the interactions between an ego and the surrounding vehicles during the procedures; and 3) we present findings from analyzing lane change procedures using real-world data on the axes of both the ego's trajectory and interactions with the scene vehicles. To the authors' knowledge, this is the first lane change behavior study from an on-road perspective that addresses the vehicle interactions in real-world traffic at the trajectory level.