Velocity control in autonomous driving is an emerging technology that has achieved rapid progress over the last decade. However, existing velocity control studies ignore cascading disturbances in multi-lane scenarios and usually ignore the negative impact caused by harsh velocity decisions. To address these issues, we propose a reinforcement learning-based framework, called RISE (contRol velocIty for autonomouSvEhicle) to make velocity decisions for an autonomous vehicle in multi-lane traffic scenarios. To detect latent disturbances in the traffic flow, we propose a novel state encoder to learn the spatio-temporal correlation between different vehicles based on a well-designed impact graph. Afterward, we introduce an actor-critic paradigm to make velocity decisions with the aid of a hybrid reward function considering four optimization objectives: safety, efficiency, comfort, and impact. In particular, the impact term can penalize the harsh decisions of the autonomous vehicle, thus encouraging it to reduce the negative impacts on traffic flow. Further, we propose an improved RISE (RISE++) framework that incorporates a motion prediction model to augment state features for reasonable decisions, a modification of the reward function for energy efficiency, and a multi-worker paradigm for training efficiency. Extensive experiments offer evidence that the proposed framework can advance the state of the art in terms of effectiveness and efficiency.

Autonomous driving is an emerging technology that has developed rapidly over the last decade, with decision-making remaining a critical challenge, particularly due to its significant role in traffic congestion. In this paper, we propose a novel perception-and-decision framework, calledHEAD, which consists of an enHanced pErception module and a mAneuverDecision module to address this challenge. In the enhanced perception module, a graph-based state prediction model with a strategy of phantom vehicle construction is proposed to address incomplete vehicle features and predict future states in parallel. Then in the maneuver decision module, a deep reinforcement learning-based model is designed to learn a driving policy based on a parameterized action Markov decision process. A hybrid reward function takes into account aspects of safety, efficiency, comfort, and impact to guide the autonomous vehicle to make optimal maneuver decisions. To make our framework applicable to more scenarios, we further propose an improvedHEAD(HEAD++) framework that makes the autonomous vehicle adapt to various road structures, such as lane merging and diverging scenarios. Besides, we develop a style tuning module inHEAD++, which supports personalized driving style tuning. To mitigate high training costs, an efficient style tuning method with approximate gradient descent is proposed to reduce the number of training iterations. Extensive experiments demonstrate the effectiveness of our framework. Compared to state-of-the-art methods,HEAD++ reduces overall traffic disturbance by 23.3%-40.9%, lowers collision risk by 4.5%-17.8%, and improves passenger comfort by 13.1%-30.5%, while maintaining high traffic efficiency.

Anomaly detection is one of the most significant tasks in industrial automatic maintenance, such as in distributed cloud systems. However, the implementation of existing anomaly detection methods is still challenging in (i) capturing the complex spatial and temporal correlations of multivariate time series, (ii) effectively adapting to the unsupervised condition, and (iii) generalizing across nodes in distributed systems. To address these challenges, we design a multi-perspective spatio-temporal attention model, called STAMP, which consists of a prediction module ST-ATTN, a reconstruction module AutoEncoder, and an adversarial optimizing module. Specifically, ST-ATTN leverages multiple attention mechanisms to perform spatio-temporal learning from both local and global perspectives, AutoEncoder is utilized to fit implicit representations, and the adversarial optimization module employs a min-max training strategy to enhance the learning capability. By introducing pre-training strategies, STAMP can be effectively adapted to distributed systems with a strong generalization ability. Furthermore, to cope with the practical unlabeled data conditions, we propose an unsupervised framework compatible with not only STAMP but also other advanced detection models. In this framework, a screening process is first conducted by traditional methods to generate a training set of pseudo-normal samples. Second, the models are trained and then used for detection. The framework can be further optimized by performing feature selection based on model-derived information for a better detectability. Extensive experiments in real-world datasets demonstrate that the proposed model and framework achieve superior performance when compared with baselines under both semi-supervised and unsupervised conditions. In particular, the detection framework has already been applied in Huawei's GaussDB (DWS) system.

Anomaly diagnosis is a fundamental problem in operation and maintenance of distributed cloud databases. Existing deep learning based methods solve this problem by classifying the anomalies with different root causes. However, since anomalies seldom occur, and anomalies with the same root cause may exhibit significantly different behaviors across different cloud database clusters, existing methods often lack sufficient training data, and they cannot generalize well from some clusters to others. Therefore, we take both anomaly and normal data into consideration, based on an observation that the discrepancy between the anomaly and normal data is relatively consistent compared to the behaviours of anomalies themselves. We design a Siamese Discrepancy Network (SDN) to learn representations of such discrepancy under the case that only a small amount of training data is available. In addition, a discrepancy-based diagnosis paradigm is proposed to construct training data for SDN and diagnose based on representations of discrepancy learned by SDN. Finally, we develop an anomaly interpretation method based on SDN, which accurately locates the symptom KPIs and root cause KPIs. Extensive experiments are conducted on both synthetic and real-world datasets. The experimental results demonstrate that the proposed method outperforms existing methods with respect to anomaly diagnosis and anomaly interpretation. In particular, the anomaly diagnosis framework has already been applied in Huawei's GaussDB (DWS) system.

In the realm of big data and cloud analytics, efficiently managing and retrieving high-dimensional data presents a critical challenge. Traditional indexes often struggle with the storage overhead inherent in large datasets. There is a growing interest in the adoption of Small Materialize Aggregation (SMA) among cloud database vendors due to its ability to maintain lightweight block-level metadata, facilitating efficient block skipping. However, SMA performance relies heavily on data layout. This is especially critical in scenarios with wide tables containing hundreds of dimensions, where the curse of dimensionality exacerbates the issue. In this paper, we propose AdaCurve , a novel approach aimed at enhancing block skipping in high-dimensional datasets through adaptive optimization of data layout. Unlike conventional static and non-adaptive space-filling curves (SFCs), AdaCurve leverages machine learning to develop an adaptive curve---a dynamically adjusting optimal projection function tailored to high-dimensional workloads and data characteristics. We introduce an attention-based network to handle high-dimensional data and a learnable objective for training adaptive curves in an end-to-end manner. Extensive experiments conducted on the Spark with real-world datasets demonstrate the effectiveness of AdaCurve . We have shown that AdaCurve effectively scales to datasets with dimensions of up to 1,000 columns, achieving a 2.8× improvement in block skipping compared to SFCs.

A series of studies apply machine learning to assist cost-based query optimizers in DBMS, emphasizing incorporating uncertainty predictions to guide decision-making. While these approaches have demonstrated advancement in some benchmarks, their drawbacks, such as unstable performance, stem from the inherent challenges of using machine learning models to predict the cost of execution plans and the lack of exploration of the intrinsic characteristics of suboptimal plans. In this paper, we introduce an alert system for query optimization, which is built upon cost models to reduce the selection of regressed plans. The key insight is that there are differences in the predictive uncertainty that lead to query optimization and the regression of execution plans. We investigate the causes of these differences in uncertainty and design a discriminator to filter out execution plans with higher risks of regression. The alert system can be integrated with various cost models, enhancing the robustness of query optimizers. In our experiments, the system further reduces execution time by 20% compared to learned optimizers. Meanwhile, the proportion of optimized queries reduced by the alert system is just 15% of the proportion of regressed queries diminished.

Searching for similar trajectories from a database is an important way for extracting human-understandable knowledge. However, due to the huge volume of trajectories and high computation complexity of distance between trajectories, it is difficult to search for exact results, which motivates the research of approximating approaches. In this study, we propose a learning to hash method for trajectory similarity computation and search, called Traj2Hash, which consists of a two-channel trajectory encoder and a hash layer to encode trajectories into Euclidean and Hamming space, respectively. The embeddings of trajectories obtained from the encoder are capable of preserving the reverse symmetric property and more representative due to the reverse augmentation and the lower-bound induced read-out layer. Moreover, we design a decomposed grid representation in the encoder to make the model lighter and better. In the model training phase, we combine a weighted mean squared error loss and a ranking-based hashing loss to enable the model similarity-aware and representations self-structured, respectively, in which a fast trajectory triplet generation method is leveraged to enrich the training corpus. Extensive experiments conducted on real data offer evidence of the effectiveness and efficiency of the proposed model.

Autonomous driving is an emerging technology that has advanced rapidly over the last decade. Modern transportation is expected to benefit greatly from a wise decision-making framework of autonomous vehicles, including the improvement of mobility and the minimization of risks and travel time. However, existing methods either ignore the complexity of environments only fitting straight roads, or ignore the impact on surrounding vehicles during optimization phases, leading to weak environmental adaptability and incomplete optimization objectives. To address these limitations, we propose a pArameterized decision-making framework with mU lti-modality percepTiOn based on deep reinforcement learning, called AUTO. We conduct a comprehensive perception to capture the state features of various traffic participants around the autonomous vehicle, based on which we design a graph-based model to learn a state representation of the multi-modal semantic features. To distinguish between lane-following and lane-changing, we decompose an action of the autonomous vehicle into a parameterized action structure that first decides whether to change lanes and then computes an exact action to execute. A hybrid reward function takes into account aspects of safety, traffic efficiency, passenger comfort, and impact to guide the framework to generate optimal actions. In addition, we design a regularization term and a multi-worker paradigm to enhance the training. Extensive experiments offer evidence that AUTO can advance state-of-the-art in terms of both macroscopic and microscopic effectiveness.

Automation in road vehicles is an emerging technology that has developed rapidly over the last decade. There have been many inter-disciplinary challenges posed on existing transportation infrastructure by autonomous vehicles. In this paper, we conduct an algorithmic study on when and how an autonomous vehicle should change its lane, which is a fundamental problem in vehicle automation field and root cause of most ‘phantom’ traffic jams. We propose a prediction-and-decision framework, called Cheetah (Changelanesmartforautonomous vehicle), which aims to optimize the lane changing maneuvers of autonomous vehicle while minimizing its impact on surrounding vehicles. In the prediction phase, Cheetah learns the spatio-temporal dynamics from historical trajectories of surrounding vehicles with a deep model (GAS-LED model) and predict their corresponding actions in the near future. A global attention mechanism and state sharing strategy are also incorporated to achieve higher accuracy and better convergence efficiency. Then in the decision phase, Cheetah looks for optimal lane change maneuvers for the autonomous vehicle by taking into account a few factors such as speed, impact on other vehicles and safety issues. A tree-based adaptive beam search algorithm is designed to reduce the search space and improve accuracy. In order to make our framework applicable to more scenarios, we further propose an improved Cheetah (Cheetah+) framework that makes the autonomous vehicle adapt for exiting a road and meet the requirement for driving comfort. Extensive experiments offer evidence that the proposed framework can advance the state of the art in terms of effectiveness and efficiency.

Autonomous driving is a rapidly advancing field that promises to revolutionize the transportation industry through an intelligent perception-and-decision paradigm. Despite decades of research, existing methods are limited in adapting to complex scenarios or expanding to unseen situations, which pose significant challenges to the development of autonomous driving. Inspired by the process of human learning to drive, autonomous vehicles can prioritize developing driving capabilities in basic scenarios and then extending the atomic abilities to more complex scenarios. To this end, we proposed a perception-and-decision framework, called ATEND, which consists of an adaptive perception module and a maneuver decision module. Specifically, the perception module based on Variational Autoencoder is proposed to map perceptual data of complex scenarios into basic scenarios. Then the reinforcement learning-based decision module can make high-level decisions in transformed scenarios. Once ATEND learns to drive in basic scenarios, it can achieve safe and efficient driving in real scenarios without additional training. Extensive experiments in different traffic scenarios evidence that the proposed framework advances the state of the art in terms of both macroscopic and microscopic effectiveness.

Autonomous driving systems (ADSs) have the potential to revolutionize transportation by improving traffic safety and efficiency. As the core component of ADSs, maneuver decision aims to make tactical decisions to accomplish road following, obstacle avoidance, and efficient driving. In this work, we consider a typical but rarely studied task, called Target-Lane-Entering (TLE), where an autonomous vehicle should enter a target lane before reaching an intersection to ensure a smooth transition to another road. For navigation-assisted autonomous driving, a maneuver decision module chooses the optimal timing to enter the target lane in each road section, thus avoiding rerouting and reducing travel time. To achieve the TLE task, we propose a ruLe-aided reINforcement lEarning framework, called LINE, which combines the advantages of RL-based policy and rule-based strategy, allowing the autonomous vehicle to make target-oriented maneuver decisions. Specifically, an RL-based policy with a hybrid reward function is able to make safe, efficient, and comfortable decisions while considering the factors of target lanes. Then a strategy of rule revision aims to help the policy learn from intervention and block the risk of missing target lanes. Extensive experiments based on the SUMO simulator confirm the effectiveness of our framework. The results show that LINE achieves state-of-the-art driving performance with over 95% task success rate.

Autonomous driving is an emerging technology that has developed rapidly over the last decade. There have been numerous interdisciplinary challenges imposed on the current transportation system by autonomous vehicles. In this paper, we conduct an algorithmic study on the autonomous vehicle decision-making process, which is a fundamental problem in the vehicle automation field and the root cause of most traffic congestion. We propose a perception-and-decision framework, called HEAD, which consists of an enHanced pErception module and a mAneuver Decision module. HEAD aims to enable the autonomous vehicle to perform safe, efficient, and comfortable maneuvers with minimal impact on other vehicles. In the enhanced perception module, a graph-based state prediction model with a strategy of phantom vehicle construction is proposed to predict the one-step future states for multiple surrounding vehicles in parallel, which deals with sensor limitations such as limited detection range and poor detection accuracy under occlusions. Then in the maneuver decision module, a deep reinforcement learning-based model is designed to learn a policy for the autonomous vehicle to perform maneuvers in continuous action space w.r.t. a parameterized action Markov decision process. A hybrid reward function takes into account aspects of safety, efficiency, comfort, and impact to guide the autonomous vehicle to make optimal maneuver decisions. Extensive experiments offer evidence that HEAD can advance the state of the art in terms of both macroscopic and microscopic effectiveness.

A reliable autonomous driving system should take safe and efficient actions in constantly changing traffic. This requires the trajectory prediction model to continuously learn from incoming data and adapt to new scenarios. In the context of rapidly growing data volume, existing trajectory prediction models must retrain on all datasets to avoid forgetting previously learned knowledge when facing additional data from new environments. In contrast, the paradigm of continual learning solely necessitates training on new data, saving a significant amount of training overhead. Therefore, it is crucial to equip the trajectory prediction model with the ability of continual learning. In this paper, inspired by rehearsal and pseudo-rehearsal methods in continual learning, we propose a continual trajectory prediction framework with uncertainty-aware generative memory replay, CTP-UGR. Our framework effectively avoids excessive memory space requirements while generating trajectory data that is authentic, representative and discriminative for continual learning. Extensive experiments on two real-world datasets demonstrate our proposed CTP-UGR significantly outperforms other baselines in terms of both accuracy and catastrophic forgetting. Besides, our framework can be combined with other state-of-the-art trajectory prediction models to achieve better performance.

Trajectory-User Linking (TUL) aiming to identify users of anonymous trajectories, has recently received increasing attention due to its wide range of applications, such as criminal investigation and personalized recommendation systems. In this paper, we propose a flexible Semi-Supervised framework for Trajectory-User Linking, namely S2TUL, which includes five components: trajectory-level graph construction, trajectory relation modeling, location-level sequential modeling, a classification layer and greedy trajectory-user relinking. The first two components are proposed to model the relationships among trajectories, in which three homogeneous graphs and two heterogeneous graphs are firstly constructed and then delivered into the graph convolutional networks for converting the discrete identities to hidden representations. Since the graph constructions are irrelevant to the corresponding users, the unlabelled trajectories can also be included in the graphs, which enables the framework to be trained in a semi-supervised way. Afterwards, the location-level sequential modeling component is designed to capture fine-grained intra-trajectory information by passing the trajectories into the sequential neural networks. Finally, these two level representations are concatenated into a classification layer to predict the user of the input trajectory. In the testing phase, a greedy trajectory-user relinking method is proposed to assure the linking results satisfy the timespan overlap constraint. We conduct extensive experiments on three public datasets with six representative competitors. The evaluation results demonstrate the effectiveness of the proposed framework.

Query optimization has long been a fundamental yet challenging topic in the database field. With the prosperity of machine learning (ML), some recent works have shown the advantages of reinforcement learning (RL) based learned query optimizer. However, they suffer from fundamental limitations due to the data-driven nature of ML. Motivated by the ML characteristics and database maturity, we propose LEON -a framework for ML-aidEd query OptimizatioN. LEON improves the expert query optimizer to self-adjust to the particular deployment by leveraging ML and the fundamental knowledge in the expert query optimizer. To train the ML model, a pairwise ranking objective is proposed, which is substantially different from the previous regression objective. To help the optimizer to escape the local minima and avoid failure, a ranking and uncertainty-based exploration strategy is proposed, which discovers the valuable plans to aid the optimizer. Furthermore, an ML model-guided pruning is proposed to increase the planning efficiency without hurting too much performance. Extensive experiments offer evidence that the proposed framework can outperform the state-of-the-art methods in terms of end-to-end latency performance, training efficiency, and stability.

Some recent works have shown the advantages of reinforcement learning (RL) based learned query optimizers. These works often use the cost (i.e., the estimation of cost model) or the latency (i.e., execution time) as guidance signals for training their learned models. However, cost-based learning underperforms in latency and latency-based learning is time-intensive. In order to bypass such a dilemma, researchers attempt to transfer a learned value network from the cost domain to the latency domain. We recognize critical insights in cost/latency-based training, prompting us to transfer the reward function rather than the value network. Based on this idea, we propose a two-stage RL-based framework, BASE , to bridge the gap between cost and latency. After learning a policy based on cost signals in its first stage, BASE formulates transferring the reward function as a variant of inverse reinforcement learning. Intuitively, BASE learns to calibrate the reward function and updates the policy regarding the calibrated one in a mutually-improved manner. Extensive experiments exhibit the superiority of BASE on two benchmark datasets: Our optimizer outperforms traditional DBMS, using 30% less training time than SOTA methods. Meanwhile, our approach can enhance the efficiency of other learning-based optimizers.

Query cost estimation, which is to estimate the query plan cost and query execution cost, is of utmost importance to query optimizers. Query plan cost estimation heavily relies on accurate cardinality estimation, and query execution cost estimation gives good hints on query latency, both of which are challenging in database management systems. Despite decades of research, existing studies either over-simplify the models only using histograms and polynomial calculation that leads to inaccurate estimates, or over-complicate them by using cumbersome neural networks with the requirements for large amounts of training data hence poor computational efficiency. Besides, most of the studies ignore the diversity of query plan structures. In this work, we propose a plan-based query cost estimation framework, called Saturn, which can eStimate cardinality and latency accurately and efficiently, for any query plan structures. Saturn first encodes each query plan tree into a compressed vector by using a traversal-based query plan autoencoder to cope with diverse plan structures. The compressed vectors can be leveraged to distinguish different query types, which is highly useful for downstream tasks. Then a pseudo label generator is designed to acquire all cardinality and latency labels with the execution part of the query plans in the training workload, which can significantly reduce the overhead of collecting the real cardinality and latency labels. Finally, a chain-wise transfer learning module is proposed to estimate the cardinality and latency of the query plan in a pipeline paradigm, which further enhances the efficiency. An extensive empirical study on benchmark data offers evidence that Saturn outperforms the state-of-the-art proposals in terms of accuracy, efficiency, and generalizability for query cost estimation.

Trajectory prediction enables the fast and accurate response of autonomous driving navigation in complex and dense traffics. In this paper, we present a novel trajectory prediction network called Heterogeneous Graph Aggregation (HeGA) for high-density heterogeneous traffic, where the traffic agents of various categories interact densely with each other. To predict the trajectory of a target agent, HeGA first automatically selects neighbors that interact with it by our proposed adaptive neighbor selector, and then aggregates their interactions based on a novel two-phase aggregation transformer block. At last, the historical residual connection LSTM enhances the historical information awareness and decodes the spatial coordinates as the prediction results. Extensive experiments on real data demonstrate that the proposed network significantly outperforms the existing state-of-the-art competitors by over 27% on average displacement error (ADE) and over 31% on final displacement error (FDE). We also deploy HeGA in a state-of-the-art framework for autonomous driving, demonstrating its superior applicability based on three simulated environments with different densities and complexities.

The ubiquity of mobile devices and the accompanying deployment of sensing technologies have resulted in a massive amount of trajectory data. One important fundamental task is trajectory similarity computation, which is to determine how similar two trajectories are. To enable effective and efficient trajectory similarity computation, we propose a novel robust model, namely Contrastive Learning based Trajectory Similarity Computation (CL-TSim). Specifically, we employ a contrastive learning mechanism to learn the latent representations of trajectories and then calculate the dissimilarity between trajectories based on these representations. Compared with sequential auto-encoders that are the mainstream deep learning architectures for trajectory similarity computation, CL-TSim does not require a decoder and step-by-step reconstruction, thus improving the training efficiency significantly. Moreover, considering the non-uniform sampling rate and noisy points in trajectories, we adopt two type of augmentations, i.e., point dowm-sampling and point distorting, to enhance the robustness of the proposed model. Extensive experiments are conducted on two widely-used real-world datasets, i.e., Porto and ChengDu, which demonstrate the superior effectiveness and efficiency of the proposed model.

Velocity control in autonomous driving is an emerging technology that has achieved rapid progress over the last decade. However, existing velocity control models are developed in single-lane scenarios and ignore the negative impacts caused by harsh velocity changes. In this work, we propose a velocity control framework based on reinforcement learning, called RISE (contRol velocIty for autonomouS vEhicle). In multi-lane circumstances, RISE improves velocity decisions regarding the autonomous vehicle itself, while minimizing impacts on rear vehicles. To achieve multiple objectives, we propose a hybrid reward function to rate each velocity decision from four aspects: safety, efficiency, comfort, and negative impact to guide the autonomous vehicle. Among these reward factors, the negative impact is used to penalize the harsh actions of the autonomous vehicle, thus prompting it to reduce the negative impacts on its rear vehicles. To detect the latent perturbations among surrounding vehicles in multiple lanes, we propose an attention-based encoder to learn the positions and interactions from an impact graph. Extensive experiments evidence that RISE enables safe driving, and outperforms state-of-the-art methods in efficiency, comfort, and alleviating negative impacts.

Hazardous chemicals transportation (HCT) brings significant financial, environmental, and health-related risks. It is imperative that a robust regulatory system is in place to reduce the risk of accidents occurring while such hazardous chemicals are being transported. Governments around the world use GPS sensors to monitor the raw trajectories of HCT trucks, but they have difficulty detecting the loaded trajectories, which is of utmost importance for the management of H CT processes. The loaded trajectory refers to the subtrajectory generated by tracking an HCT truck when it is loaded with hazardous chemical in an HCT process. The stay points in the raw trajectory provide some feasibility to detect the loaded trajectory as they reflect the potential loading and unloading actions of the HCT truck. However, directly using the stay points to detect the loaded trajectory usually leads to unsatisfactory results due to two chal-lenges: (1) complex staying scenarios, and (2) numerous loading and unloading locations. To tackle the challenges, we propose a LoadEd trAjectory Detectlon framework, called LEAD, to detect the loaded trajectory from the raw HCT trajectory accurately and efficiently. LEAD processes a raw trajectory into a set of candidate trajectories, encodes each candidate trajectory into a latent representation, and detects the loaded trajectory using the latent representations of candidate trajectories. Extensive experiments based on a real-world dataset from Nantong, China confirm the effectiveness of our framework. The results show that the detection accuracy of LEAD exceeds 83 % which outperforms competing baselines by over 42 %.

Join order selection plays an important role in DBMS query optimizers. The problem aims to find the optimal join order with the minimum cost, and usually becomes an NP-hard problem due to the exponentially increasing search space. Recent advanced studies attempt to use deep reinforcement learning (DRL) to generate better join plans than the ones provided by conventional query optimizers. However, DRL-based methods require time-consuming training, which is not suitable for online applications that need frequent periodic re-training. In this paper, we propose a novel framework, namely efficient Join Order selection learninG with Graph-basEd Representation (JOGGER). We firstly construct a schema graph based on the primary-foreign key relationships, from which table representations are well learned to capture the correlations between tables. The second component is the state representation, where a graph convolutional network is utilized to encode the query graph and a tailored-tree-based attention module is designed to encode the join plan. To speed up the convergence of DRL training process, we exploit the idea of curriculum learning, in which queries are incrementally added into the training set according to the level of difficulties. We conduct extensive experiments on JOB and TPC-H datasets, which demonstrate the effectiveness and efficiency of the proposed solutions.

The largest portion of urban congestion is caused by 'phantom' traffic jams, causing significant delay travel time, fuel waste, and air pollution. It frequently occurs in high-density traffics without any obvious signs of accidents or roadworks. The root cause of 'phantom' traffic jams in one-lane traffics is the sudden change in velocity of some vehicles (i.e. harsh driving behavior (HDB)), which may generate a chain reaction with accumulated impact throughout the vehicles along the lane. This paper makes the first attempt to address this notorious problem in a one-lane traffic environment through velocity control of autonomous vehicles. Specifically, we propose a velocity control framework, called PATROL (sPAtial-temporal ReinfOrcement Learning). First, we design a spatial-temporal graph inside the reinforcement learning model to process and extract the information (e.g. velocity and distance difference) of multiple vehicles ahead across several historical time steps in the interactive environment. Then, we propose an attention mechanism to characterize the vehicle interactions and an LSTM structure to understand the vehicles' driving patterns through time. At last, we modify the reward function used in previous velocity control works to enable the autonomous driving agent to predict the HDB of preceding vehicles and smoothly adjust its velocity, which could alleviate the chain reaction caused by HDB. We conduct extensive experiments to demonstrate the effectiveness and superiority of PATROL in alleviating the 'phantom' traffic jam in simulation environments. Further, on the real-world velocity control dataset, our method significantly outperforms the existing methods in terms of driving safety, comfortability, and efficiency.

Automation in road vehicles is an emerging technology that has developed rapidly over the last decade. There have been many inter-disciplinary challenges posed on existing transportation infrastructure by autonomous vehicles (AV). In this paper, we conduct an algorithmic study on when and how an autonomous vehicle should change its lane, which is a fundamental problem in vehicle automation field and root cause of most 'phantom' traffic jams. We propose a prediction-and-search framework, called Cheetah (Change lane smart for autonomous vehicle), which aims to optimize the lane changing maneuvers of autonomous vehicle while minimizing its impact on surrounding vehicles. In the prediction phase, Cheetah learns the spatio-temporal dynamics from historical trajectories of surrounding vehicles with a deep model (GAS-LED) and predict their corresponding actions in the near future. A global attention mechanism and state sharing strategy are also incorporated to achieve higher accuracy and better convergence efficiency. Then in the search phase, Cheetah looks for optimal lane change maneuvers for the autonomous vehicle by taking into account a few factors such as speed, impact on other vehicles and safety issues. A tree-based adaptive beam search algorithm is designed to reduce the search space and improve accuracy. Extensive experiments on real and synthetic data evidence that the proposed framework excels state-of-the-art competitors with respect to both effectiveness and efficiency.