Data-driven approaches have been recently proven effective in pedestrian action prediction by extensive works. However, frame-level annotations of pedestrian actions require a significant amount of manpower and time. In this paper, we propose a simple yet effective contrastive learning framework that enables pedestrian action prediction models to be trained on data without action labels. First of all, we regard disentangled visual observations, such as appearance, motion and trajectories, as multiple modalities. Then we construct a joint latent space where multimodal features from the same sample are encouraged to be close, whereas features from different samples are encouraged to be far from each other. Since most existing models use a similar architecture composed of separate feature extractors and fusion modules, our proposed framework can be applied directly to existing methods to boost the feature extractors. We pretrained state-of-the-art models on datasets without action labels, nuScenes and BDD100k, and evaluated these models on PIE, JAAD and TITAN. Quantitative results show that the pretrained with only the fusion parameters fine-tuned can compete with or even outperform models that are completely trained the one dataset.

Currently, with the rapid growth of training datasets for autonomous driving systems, we are faced with a challenge: how to efficiently retrieve specific traffic scenes from massive amount of scene in multiple datasets. This challenge primarily stems from the heterogeneity of existing datasets, meaning these datasets contain different types of data, follow different data formats, and use different sensors for data collection. To address this issue, we present RSG-Search Plus, a universal traffic scene searching method based on Road Scene Graph and Large Language Models (LLMs). Our approach first transform datasets into scene graphs to exclude irrelevant details, then efficiently retrieving specific configurations among thousands of traffic scenes by matching isomorphic sub-graphs between input graph and road scene graph. Experimental results demonstrate that our graph searching method can accurately match the scenes described by input condition. Additionally, this method is easily adaptable to different datasets, significantly simplifying the scene search process.

Nowadays, using LiDARs, RGB cameras and Thermal cameras for automatic systems, in particular self-driving cars, has become the common approach in multiple deployments. Each kind of sensor has distinct advantages, leading to the fact that using multiple sensors can help autonomous systems improve their performance. Calibration between sensors is the precondition of fusing multiple sensors. This paper presents a novel way to register extrinsic parameters for LiDAR, 360 RGB camera and 360 Thermal camera automatically based on features information. To evaluate the method, we use our dataset around Nagoya University.

Cooperative recognition, a method to achieve human-automation cooperation in the recognition phase of the autonomous driving system, has been proposed to address the challenges in the conventional control phase cooperation, e.g., taking over vehicle control. In cooperative recognition, the operator intervenes in recognition tasks that are difficult for the automated system alone to improve driving efficiency and safety. The challenge is the integration of both human and automated systems while both participants have different characteristics, processing capabilities, and uncertainty in the decisions (recognition results). The objectives of this study are task allocation (i.e., when and for which targets the operator should intervene) taking into account the intervention efficiency and human state. And also combine the human intervention and recognition result of the automated systems to solve the uncertainties in both participants. We formulated this problem with a Partially Observable Markov Decision Process (POMDP). The simulator experiment indicated that the recognition result of the automated system and the operator’s intervention were stochastically combined. The intervention requests to the operator adapted to the operator state and could be reduced while maintaining driving efficiency and minimizing risk omissions.

Semantic segmentation aims at assigning labels to every pixel of a given image. In the context of autonomous vehicles, semantic segmentation models should be trained with data collected from the traffic network through which vehicles are expected to circulate. Road regulation, weather conditions, and other context features may differ between regions, making local semantic segmentation datasets extremely valuable. However, the high ground truth annotation costs represent a hindrance to the development of such models. The upsurge of powerful feature learning architectures leaves room for semantic segmentation models trained on an unsupervised fashion. This observation vertebrates the purpose of this work: to produce coarse segmentation maps for scene understanding without the need of annotated data. We depart from an unsupervised model that yields low-quality results. The proposed methodology establishes a set of guidelines for the enhancement of segmentation maps. Obtained results expose an improvement of the segmentation quality thanks to the application of our devised guidelines, paving the way for the automatic generation of semantic segmentation datasets.

Autonomous driving of vehicles and robots requires highly accurate position information, and RTK-GNSS is expected to be utilized for this purpose. In this paper, we propose a robust and real-time operation method by introducing graph optimization into the integrated RTK-GNSS/IMU method. The proposed method is an extension of a method using vehicle trajectories that can estimate positions with lane-level accuracy even in urban areas. The position is estimated by removing GNSS multipaths from the shape of a vehicle trajectory of several hundred meters and averaging the remaining GNSS results. This method does not take into account the errors in the vehicle trajectory and cannot fully benefit from the high accuracy positioning solution of RTK-GNSS. To solve this problem, we introduce graph optimization to the base method, which treats the error state as a probabilistic model. However, general graph optimization methods have problems with processing time and outlier elimination. The proposed method solves these problems by restricting the time series data to be optimized and using a two-step optimization structure. Evaluations show that the proposed method is effective because it satisfies the requirements for real-time operation and improves accuracy compared to conventional methods.

Browsing specific traffic scene in large-scale dataset is an increasing demand from researchers, self-driving community and insurance companies. It is easy to search scenes with specific tags such as "rain", "snow", or "on highway". However, searching specific scene configurations, like "two vehicles waiting for a person crossing the road", is still an open problem. In this paper, we provide RSG-search, a scene-graph based traffic scene retrieval method, based on our previous research on traffic scene-graph generation. By previously translating open datasets to scene graphs, we can ignore irrelevant details, and efficiently search specific scene configuration among thousands of traffic scenes. Experiment results shows that our graph searching method is able to retrieve results for a given query with high accuracy. Our method simplifies the task of scene retrieval, opening opportunities for new applications.

Intelligent vehicle perception algorithms often have difficulty accurately analyzing and interpreting images in adverse weather conditions. Snow is a corner case that not only reduces visibility and contrast but also affects the stability of the road environment. While it is possible to train deep learning models on real-world driving datasets in snow weather, obtaining such data can be challenging. Synthesizing snow effects on existing driving datasets is a viable alternative. In this work, we propose a method based on Cycle Consistent Generative Adversarial Networks (CycleGANs) that utilizes additional semantic information to generate snow effects. We apply deep supervision by using intermediate outputs from the last two convolutional layers in the generator as multi-scale supervision signals for training. We collect a small set of driving image data captured under heavy snow as the translation source. We compare the generated images with those produced by various network architectures and evaluate the results qualitatively and quantitatively on the Cityscapes and EuroCity Persons datasets. Experiment results indicate that our model can synthesize realistic snow effects in driving images.

Snow corrupts LiDAR point clouds with scattered noise points and false objects, posing a serious threat to the perception of autonomous driving systems. Existing effective point cloud de-snow methods are mainly based on outlier filters that rigidly remove isolated points. There are deep-learning and algorithm-based weather models that can handle adverse conditions such as rain and fog, but snow conditions are rarely considered. In this study, we propose a LiDAR point cloud translation model based on refined generative adversarial networks (GANs) that is not only able to de-noise snow in point clouds but also to generate fake snow points on clear data. Our model is trained on depth image representations of point clouds from unpaired datasets, with a customized loss function for grayscale depth images that can maintain scale consistency. A pixel-wise discriminator structure is designed to improve the de-snowing effect around the ego vehicle. The proposed model expresses a better feature capture on snow in LiDAR point clouds, and experiment results show high-quality snow removal performance on both the scattered and clustered snow points, as well as satisfactory fake snow generation on clear road point clouds.

When developing truly driverless mobility for the future, one key index used to measure the matureness of a particular self-driving technology is the driver intervention rate. One method which has proven to be effective for decreasing intervention rates is the use of personalized driving models that can mimic the driving style and preferences of a targeted user, so that autonomous driving feels safer and more natural to them. To create such models, quantitative data should be collected from users in order to determine the style of driving that a particular user, or type of user, prefers. In this paper, we introduce the Subjective Risk Lane Change (SRLC) Dataset, which includes ego vehicle driving behavior data, surrounding vehicle location information, and the subjective risk scores of users, collected during both safe and risky lane change scenarios encountered in CARLA simulators, as well as demographic information for our 30 participants. Furthermore, user intervention data for all of our participants was collected from Personalized Model Predictive Controllers during the generated lane change maneuvers. As far as the authors are able to determine, no other public dataset provides driving behavior signal and intervention timing information collected during driver interventions. Our dataset can be used to gain insights into a variety of personal driving styles, allowing the improvement of adaptive autonomous driving systems, and leading to safer and more widely accepted driverless technology.

Reproducing real-world traffic scenes in the simulator is fundamental to training self-driving systems. Creating a simulation scenario is a complex task, generally done manually: the ego-vehicle and other entities are placed and their trajectories defined, trying to recreate some situation found in real traffic. To reduce the manual burden, here we propose the Real-to-Synthetic toolset. This toolset provides synthetic traffic scene in openDrive format, which can be directly simulated in many simulators such as SUMO or CARLA. Also, we provide a scene generator which generates near-realistic scene from minimum user effort. To maintain the similarity between real-world scene and generated one, here we introduce the concept “Road Scene Graph”(RSG). In this graph, nodes represent entities while edges stand for pairwise relationships. These relationships could be maintained in the scene generation process while the actor is generated according to the distribution sampled from real-world data. Experiments proved that by using “Road Scene Graph”, our scene generator proposes a much more convenient way to conFigure traffic scenes rather than manually defining every actor’s initial status and trajectories.

Bad weather such as rain and haze will lower the visibility and contrast of captured images and often occur at the same time, which makes the situation worse. Existing CNN-based methods can achieve impressive results when processing each condition individually. But few works consider removing rain and haze under a unified framework. Besides, these models will experience degraded performance when the distribution of the data changes. In this work, we attempt to find a method that can remove all bad weather without switching between single task models and paired training data for driving scene applications like pedestrian detection. In specific, we adopt a disentanglement strategy to obtain weather layer from subtraction. The statistic distance is calculated between two weather layers from each pipeline. In addition, the weather layer serves as information guidance and input to the reconstruct generator. Experiments on mixed dataset from RainCityscapes and Foggy Cityscapes show that the effectiveness of pedestrian detector is improved. Finally, we contribute a new dataset called Realistic Driving Scene under Bad Weather (RDSBW), which contains over 70K real bad weather images.

Human drivers have complex and unique driving characteristics, even when driving in common, well-defined scenarios such as lane changes. In this study, we propose using probabilistic, deep generative models to predict personalized driving behavior including velocity, acceleration, and steering angle sequence. Probabilistic approaches are applied to model uncertainty in the driving behavior of individual drivers as a distribution, while surrounding vehicle and driver ID information are considered as given conditions in the distribution. We train individual driver models using real-world driving data, and use them to predict sequences of future driving behavior in dynamic environments, using historical data to take personal driving styles into account. Our results show that the proposed driver behavior modeling method is able to learn from a driver's vehicle operation data and their interactions with surrounding vehicles to reproduce their specific driving style.

As research and development related to the practical use of autonomous driving systems (ADS) continue to advance, one of the remaining challenges is achieving both safe and natural autonomous driving. In part, this is due to the difficulty of achieving flawless, automated perception and understanding of the driving environment. In our previous study, we proposed a recognition assistance interface to solve this problem by sharing ADS recognition information with the passenger, allowing them to assist in the recognition stage of the autonomous driving process. In this study, we incorporate our recognition assistance interface into Autoware and test it in a simulated driving environment, using scenarios in which the ADS must recognize the intent of pedestrians and respond to the presence of trash in the road. Natural driving was dened as driving that avoids significant, unnecessary deceleration when performing these challenging recognition tasks. The results of our experiment with 11 participants showed that sharing recognition information with passengers is effective for avoiding unnecessary deceleration and achieving only minor variations in speed when encountering obstacles flagged in error by the recognition system.

Behavioral and semantic relationships play a vital role on intelligent self-driving vehicles and ADAS systems. Different from other research focused on trajectory, position, and bounding boxes, relationship data provides a human understandable description of the object's behavior, and it could describe an object's past and future status in an amazingly brief way. Therefore it is a fundamental method for tasks such as risk detection, environment understanding, and decision making. In this paper, we propose RSG-Net (Road Scene Graph Net): a graph convolutional network designed to predict potential semantic relationships from object proposals, and to produce a graph-structured result, called “Road Scene Graph”. The experimental results indicate that this network, trained on Road Scene Graph dataset, could efficiently predict potential semantic relationships among objects around the ego-vehicle.

Continuous point cloud data is being used more and more widely in practical applications such as mapping, localization and object detection in autonomous driving systems, but due to the huge volume of data involved, sharing and storing this data is currently expensive and difficult. One possible solution is the development of more efficient methods of compressing the data. Other researchers have proposed converting 3D point cloud data into 2D images, or using tree structures to store the data. In a previous study targeting streaming point cloud data, we proposed an MPEG-like compression method which utilizes simultaneous localization and mapping (SLAM) results to simulate LiDAR’s operating process. In this paper, instead of imitating MPEG, we propose new strategy for more efficient reference frame distribution and more natural frame prediction, and use a different algorithm to encode the residual, greatly improving the algorithm’s performance and its stability in different scenarios. We also discuss how various parameters affect compression performance. Using our proposed method, streaming point cloud data collected by LiDAR sensors can be compressed to 1/50th of its original size, with only 2 cm of Root Mean Square Error for each detected point. We evaluate our proposed method by comparing its performance with several other existing point cloud compression methods in three different driving scenarios, demonstrating that our proposed method outperforms them.

The ability to follow a human is necessary for many types of service and social robots. A development of human-following robots requires taking into account the features of a perception of such robots by users. This paper presents results of human-robot interaction experiments with TIAGo Base cargo robot when it follows a user. Eight people took part in the experiments. The participants interacted with the robot using voice commands. TIAGo Base tracked a person using a single laser rangefinder. The purpose of the experiments was to identify the features of a user interaction with TIAGo Base robot through voice commands when the robot carries a load and follows the user. In addition, we identified a preferable distance between TIAGo Base robot and a user.

In this work, we present LIBRE: LiDAR Benchmarking and Reference, a first-of-its-kind dataset featuring 10 different LiDAR sensors, covering a range of manufacturers, models, and laser configurations. Data captured independently from each sensor includes three different environments and configurations: static targets, where objects were placed at known distances and measured from a fixed position within a controlled environment; adverse weather, where static obstacles were measured from a moving vehicle, captured in a weather chamber where LiDARs were exposed to different conditions (fog, rain, strong light); and finally, dynamic traffic, where dynamic objects were captured from a vehicle driven on public urban roads, multiple times at different times of the day, and including supporting sensors such as cameras, infrared imaging, and odometry devices. LIBRE will contribute to the research community to (1) provide a means for a fair comparison of currently available LiDARs, and (2) facilitate the improvement of existing self-driving vehicles and robotics-related software, in terms of development and tuning of LiDAR-based perception algorithms.

Teaching autonomous vehicles to imitate human driving in complex, urban traffic scenarios is a difficult task. “End-to-end” autonomous driving systems, based on “imitation learning”, are an expecting approach. A model learns the relationships between sensing input and vehicle control signal outputs. These methods can successfully achieve driving in simple scenarios such as lane keeping. In contrast, the “mid-to-mid” autonomous driving methods now being proposed. In such framework, the model learns the relationships between pre-processed feature maps from the model-based system as input and the future position of the ego vehicle as the output. Mid-to-mid driving methods can direct vehicles more robustly than end-to-end driving methods in some complex driving environments. However, mid-to-mid driving methods use the results of the object detection module to create the feature map. If object detection fails, or detection performance is poor due to changes in the driving environment, prediction performance may also be degraded. Our proposed method uses a prediction module that outputs point grid maps directly, without the use of an object detection module, which are then incorporated into the feature map. Point grid maps represent the locations of surrounding vehicles and obstacles directly, based on LiDAR point cloud data. Since the results of object detection are not used by the prediction module, detection performance does not affect prediction performance. In this study we conduct two experiments, an off-line evaluation using a Lyft dataset, and an on-line evaluation using the CARLA simulator. The results show that our model can achieve the same level of ego-vehicle position prediction performance as a model using annotated object location information.

The use of 3D LiDAR, which has proven its capabilities in autonomous driving systems, is now expanding into many other fields. The sharing and transmission of point cloud data from 3D LiDAR sensors has broad application prospects in robotics. However, due to the sparseness and disorderly nature of this data, it is difficult to compress it directly into a very low volume. A potential solution is utilizing raw LiDAR data. We can rearrange the raw data from each frame losslessly in a 2D matrix, making the data compact and orderly. Due to the special structure of 3D LiDAR data, the texture of the 2D matrix is irregular, in contrast to 2D matrices of camera images. In order to compress this raw, 2D formatted LiDAR data efficiently, in this paper we propose a method which uses a recurrent neural network and residual blocks to progressively compress one frame's information from 3D LiDAR. Compared to our previous image compression based method and generic octree point cloud compression method, the proposed approach needs much less volume while giving the same decompression accuracy. Potential application scenarios for point cloud compression are also considered in this paper. We describe how decompressed point cloud data can be used with SLAM (simultaneous localization and mapping) as well as for localization using a given map, illustrating potential uses of the proposed method in real robotics applications.

Effective detection of people is a basic requirement for robot coexistence in human environments. In our previous work [1] we proposed a method for people detection and position estimation using multiple layers of Laser Range Finders (LRF) in a mobile robot. We extend our work by introducing laser reflection intensity as a novel feature for people detection, achieving significant improvement of detection rates. In concrete, we propose a method for calibration of laser intensity data, a method for segment separation using laser intensity, and introduce two new intensity-based features for people detection: the variance of laser intensity and the variance of intensity differences. We present experimental results that confirm the effectiveness of our multi-layered detection method including laser intensity.

Successful detection of people is a basic requirement for a robot to achieve symbiosis in people's daily life. Specifically, a mobile robot designed to follow people needs to keep track of people's position through time, for it defines the robot's position and trajectory.

This paper presents and describes the approach for achieving long distance autonomous navigation with a mobile robot on outdoor cluttered pedestrian paths. The task was to finish an event launched by the City of Tsukuba in Japan, called “RealWorld Robot Challenge”, of navigating 1km autonomously in a real environment with real pedestrians and bicycles. The hardware, software and strategy for navigating in cluttered environments is explained. Moreover, the complementary functionality of the overall system where map-based and sensor-based navigation seamlessly change, is presented. The robustness of the system is validated with experimental results.