Modeling complicated interactions among the ego-vehicle, road agents, and map elements has been a crucial part for safety-critical autonomous driving. Previous work on end-to-end autonomous driving relies on the attention mechanism to handle heterogeneous interactions, which fails to capture geometric priors and is also computationally intensive. In this paper, we propose the Interaction Scene Graph (ISG) as a unified method to model the interactions among the ego-vehicle, road agents, and map elements. With the representation of the ISG, the driving agents aggregate essential information from the most influential elements, including the road agents with potential collisions and the map elements to follow. Since a mass of unnecessary interactions are omitted, the more efficient scene-graph-based framework is able to focus on indispensable connections and leads to better performance. We evaluate the proposed method for end-to-end autonomous driving on the nuScenes dataset. Compared with strong baselines, our method significantly outperforms in full-stack driving tasks.

Motion prediction from raw LiDAR sensor data has drawn increasing attention and led to a surge of studies following two main paradigms. One paradigm is global motion paradigm, which simultaneously detects objects from point clouds and predicts the trajectories of each object in the future. The other paradigm is local motion paradigm, which directly performs dense motion prediction pointwisely. We observe that global motion prediction can benefit from local motion representation, since it contains rich local displacement contexts that are not explicitly exploited in global motion prediction. Correspondingly, local motion prediction can benefit from global motion representation, since it provides object contexts to improve prediction consistency inside an object. However, the complement of these two motion representations has not fully explored in the literature. To this end, we propose Hybrid Motion Representation Learning (HyMo), a unified framework to address the problem of motion prediction by making the best of both global and local motion cues. We have conducted extensive experiments on nuScenes dataset. The experimental results demonstrate that the learned hybrid motion representation achieves state-of-the-art performance on both global and local motion prediction tasks.

Trajectory prediction of multiple agents in a crowded scene is an essential component in many applications, including intelligent monitoring, autonomous robotics, and self-driving cars. Accurate agent trajectory prediction remains a significant challenge because of the complex dynamic interactions among the agents and between them and the surrounding scene. To address the challenge, we propose a decoupled attention-based spatial-temporal modeling strategy in the proposed trajectory prediction method. The past and current interactions among agents are dynamically and adaptively summarized by two separate attention-based networks and have proven powerful in improving the prediction accuracy. Moreover, it is optional in the proposed method to make use of the road map and the plan of the ego-agent for scene-compliant and accurate predictions. The road map feature is efficiently extracted by a convolutional neural network, and the features of the ego-agent’s plan is extracted by a gated recurrent network with an attention module based on the temporal characteristic. Experiments on benchmark trajectory prediction datasets demonstrate that the proposed method is effective when the ego-agent plan and the the surrounding scene information are provided and achieves state-of-the-art performance with only the observed trajectories.

Trajectory prediction is a fundamental task in many real applications such as autonomous robotics and video surveillance. In this paper, we propose a novel vision-based trajectory prediction method which is able to extract the interactive features by active global interaction-aware drivable space learning. The learned global interaction-aware drivable spaces denote the areas with low occupation probabilities, which provide the regions and directions that the agents can move into. Specifically, our method describes a sequence of motion states, i.e. the location, the velocity and the acceleration, as occupancy grid maps, and then use them to train the deep learning model in the supervised manner. Moreover, an interactive loss for training the inference net of drivable spaces and trajectory prediction net simultaneously is introduced. Comparisons with state-of-the-art methods on benchmark datasets demonstrate the effectiveness of the proposed method.

Motion prediction of multiple agents in a dynamic scene is a crucial component in many real applications, including intelligent monitoring and autonomous driving. Due to the complex interactions among the agents and their interactions with the surrounding scene, accurate trajectory prediction is still a great challenge. In this paper, we propose a new method for robust trajectory prediction of multiple intelligent agents in a dynamic scene. The input of the method includes the observed trajectories of all agents, and optionally, the planning of the ego-agent and the surrounding high definition map at every time steps. Given observed trajectories, an efficient approach in a star computational topology is utilized to compute both the spatiotemporal interaction features and the current interaction features between the agents, where the time complexity scales linearly to the number of agents. Moreover, on an autonomous vehicle, the proposed prediction method can make use of the planning of ego-agent to improve the modeling of the interaction between surrounding agents. To increase the robustness to upstream perception noises, at the training stage, we randomly mask out the input data, a.k.a. the points on the observed trajectories of agents and the lane sequence. Experiments on autonomous driving and pedestrian-walking datasets demonstrate that the proposed method is not only effective when the planning of ego-agent and the high definition map are provided, but also achieves state-of-the-art performance with only the observed trajectories.

Pedestrian trajectory prediction is crucial for many important applications. This problem is a great challenge because of complicated interactions among pedestrians. Previous methods model only the pairwise interactions between pedestrians, which not only oversimplifies the interactions among pedestrians but also is computationally inefficient. In this paper, we propose a novel model StarNet to deal with these issues. StarNet has a star topology which includes a unique hub network and multiple host networks. The hub network takes observed trajectories of all pedestrians to produce a comprehensive description of the interpersonal interactions. Then the host networks, each of which corresponds to one pedestrian, consult the description and predict future trajectories. The star topology gives StarNet two advantages over conventional models. First, StarNet is able to consider the collective influence among all pedestrians in the hub network, making more accurate predictions. Second, StarNet is computationally efficient since the number of host network is linear to the number of pedestrians. Experiments on multiple public datasets demonstrate that StarNet outperforms multiple state-of-the-arts by a large margin in terms of both accuracy and efficiency.

Face Parsing assigns every pixel in a facial image with a semantic label, which could be applied in various applications including face recognition, facial beautification, affective computing and animation. While lots of progress have been made in this field, current state-of-the-art methods still fail to extract real effective feature and restore accurate score map, especially for those facial parts which have large variations of deformation and fairly similar appearance, e.g. mouth, eyes and thin eyebrows. In this paper, we propose a novel pixel-wise face parsing method called Residual Encoder Decoder Network (RED-Net), which combines a feature-rich encoder-decoder framework with adaptive prior mechanism. Our encoder-decoder framework extracts feature with ResNet and decodes the feature by elaborately fusing the residual architectures in to deconvolution. This framework learns more effective feature comparing to that learnt by decoding with interpolation or classic deconvolution operations. To overcome the appearance ambiguity between facial parts, an adaptive prior mechanism is proposed in term of the decoder prediction confidence, allowing refining the final result. The experimental results on two public datasets demonstrate that our method outperforms the state-of-the-arts significantly, achieving improvements of F-measure from 0.854 to 0.905 on Helen dataset, and pixel accuracy from 95.12% to 97.59% on the LFW dataset. In particular, convincing qualitative examples show that our method parses eye, eyebrow, and lip regins more accurately.