In interactions between automated vehicles (AVs) and crossing pedestrians, modeling implicit vehicle communication is crucial. In this work, we present a combined prediction and planning approach that allows to consider the influence of the planned vehicle behavior on a pedestrian and predict a pedestrian's reaction. We plan the behavior by solving two consecutive optimal control problems (OCPs) analytically, using variational calculus. We perform a validation step that assesses whether the planned vehicle behavior is adequate to trigger a certain pedestrian reaction, which accounts for the closed-loop characteristics of prediction and planning influencing each other. In this step, we model the influence of the planned vehicle behavior on the pedestrian using a probabilistic behavior acceptance model that returns an estimate for the crossing probability. The probabilistic modeling of the pedestrian reaction facilitates considering the pedestrian's costs, thereby improving cooperative behavior planning. We demonstrate the performance of the proposed approach in simulated vehicle-pedestrian interactions with varying initial settings and high-light the decision making capabilities of the planning approach.

Lane-changing maneuvers, particularly those executed abruptly or in risky situations, are a significant cause of road traffic accidents. However, current research mainly focuses on predicting safe lane changes. Furthermore, existing accident datasets are often based on images only and lack comprehensive sensory data. In this work, we focus on predicting risky lane changes using the CARLA Risky-lane-change Anticipation in Simulated Highways (CRASH) dataset (our own collected dataset specifically for risky lane changes), and safe lane changes (using the HighD dataset). Then, we leverage Knowledge Graphs (KGs) and Bayesian inference to predict these maneuvers using linguistic contextual information, enhancing the model's interpretability and transparency. The model achieved a 91.5% f1-score with anticipation time extending to four seconds for risky lane changes, and a 90.0% f1-score for predicting safe lane changes with the same anticipation time. We validate our model by integrating it into a vehicle within the CARLA simulator in scenarios that involve risky lane changes. The model managed to anticipate sudden lane changes, thus providing automated vehicles with further time to plan and execute appropriate safe reactions. Finally, to enhance the explainability of our model, we utilize Retrieval Augmented Generation (RAG) to provide clear and natural language explanations for the given prediction.

Interaction modeling plays a huge role in understanding human behavior in traffic. This is especially relevant when it comes to interactions between vehicles and vulnerable road users such as pedestrians. Thus, pedestrian intention prediction is an ongoing field of research in order to understand the pedestrians' decision making. Most state-of-the-art prediction frameworks are trained on large-scale datasets and evaluated with respect to acknowledged benchmarks. These datasets lack the ability to account for the reciprocal nature of interactions between pedestrians and vehicles and the effects of the two agents influencing each other. In this work, we demonstrate first steps towards assessing pedestrian prediction algorithms within realistic scenarios including the interaction effects arising from its interplay with a planning component. For this, we validate an existing prediction framework trained on benchmark datasets with situations from a virtual reality (VR) pedestrian-vehicle co-simulator that allows us to include the effect of vehicle planning on pedestrian behavior. We evaluate the performance of the prediction framework comparing data from pre-recorded real-world datasets with data from our co-simulation study and conduct an ablation analysis to identify the most important features for pedestrian intention prediction. The results highlight the significance of pedestrian action and proximity to the road.

Pedestrian detection is a critical task in autonomous driving, aimed at improving safety and reducing risks on the road. In recent years, significant advancements have been made in detection performance. However, these achievements still fall short of human perception, particularly in cases involving occluded pedestrians, especially those entirely invisible. In this work, we present the Occlusion-Rich Road Scenes with Pedestrians (OccluRoads) Dataset, a diverse collection of road scenes with partially and fully occluded pedestrians in both real-world and virtual environments. All scenes are meticulously labeled and enriched with contextual information that encapsulates human perception in such scenarios. Leveraging this Dataset, we developed a pipeline to predict the presence of occluded pedestrians using Knowledge Graph (KG), Knowledge Graph Embedding (KGE), and a Bayesian inference process. Our approach achieves an F1 score of 0.91, representing an improvement of up to 42% compared to traditional machine learning models.

The prediction of road user behaviors in the context of autonomous driving has attracted considerable attention from the scientific community in recent years. Most works focus on predicting behaviors based on kinematic information alone, a simplification of reality since road users are humans, and as such they are highly influenced by their surrounding context. In addition, a large plethora of research works rely on powerful Deep Learning techniques, which exhibit high-performance metrics in prediction tasks but may lack the ability to fully understand and exploit the contextual semantic information contained in the road scene, not to mention their inability to provide explainable predictions that can be understood by humans. In this work, we propose an explainable road users’ behavior prediction system that integrates the reasoning abilities of Knowledge Graphs (KG) and the expressiveness capabilities of Large Language Models (LLM) by using Retrieval Augmented Generation (RAG) techniques. For that purpose, Knowledge Graph Embeddings (KGE) and Bayesian inference are combined to allow the deployment of a fully inductive reasoning system that enables the issuing of predictions that rely on legacy information contained in the graph, as well as on current evidence gathered in real-time by onboard sensors. Two use cases have been implemented following the proposed approach: 1) Prediction of pedestrians’ crossing actions; and 2) Prediction of lane change maneuvers. In both cases, the performance attained exceeds the current state-of-the-art in terms of anticipation and F1 score, showing a promising avenue for future research in this field.

This study presents the outcomes of empirical investigations pertaining to human-vehicle interactions involving an autonomous vehicle (AV) equipped with both internal and external Human Machine Interfaces (HMIs) within a crosswalk scenario. The internal and external HMIs were integrated with implicit communication techniques, incorporating a combination of gentle and aggressive braking manoeuvres within the crosswalk. Data were collected through a combination of questionnaires and quantifiable metrics, including pedestrian decision to cross related to the vehicle distance and speed. The questionnaire responses reveal that pedestrians experience enhanced safety perceptions when the external HMI and gentle braking manoeuvres are used in tandem. In contrast, the measured variables demonstrate that the external HMI proves effective when complemented by the gentle braking manoeuvre. Furthermore, the questionnaire results highlight that the internal HMI enhances passenger confidence only when paired with the aggressive braking manoeuvre.

In the field of autonomous driving research, the use of immersive virtual reality (VR) techniques is widespread to enable a variety of studies under safe and controlled conditions. However, this methodology is only valid and consistent if the conduct of participants in the simulated setting mirrors their actions in an actual environment. In this paper, we present a first and innovative approach to evaluating what we term the behavioural gap, a concept that captures the disparity in a participant’s conduct when engaging in a VR experiment compared to an equivalent real-world situation. To this end, we developed a digital twin of a pre-existed crosswalk and carried out a field experiment (N = 18) to investigate pedestrian-autonomous vehicle interaction in both real and simulated driving conditions. In the experiment, the pedestrian attempts to cross the road in the presence of different driving styles and an external Human-Machine Interface (eHMI). By combining survey-based and behavioural analysis methodologies, we develop a quantitative approach to empirically assess the behavioural gap, as a mechanism to validate data obtained from real subjects interacting in a simulated VR-based environment. Results show that participants are more cautious and curious in VR, affecting their speed and decisions, and that VR interfaces significantly influence their actions.

Prediction of vehicle lane change maneuvers has gained a lot of momentum in the last few years. Some recent works focus on predicting a vehicle’s intention by predicting its trajectory first. This is not enough, as it ignores the context of the scene and the state of the surrounding vehicles (as they might be risky to the target vehicle). Other works assessed the risk made by the surrounding vehicles only by considering their existence around the target vehicle, or by considering the distance and relative velocities between them and the target vehicle as two separate numerical features. In this work, we propose a solution that leverages Knowledge Graphs (KGs) to anticipate lane changes based on linguistic contextual information in a way that goes well beyond the capabilities of current perception systems. Our solution takes the Time To Collision (TTC) with surrounding vehicles as input to assess the risk on the target vehicle. Moreover, our KG is trained on the HighD dataset using the TransE model to obtain the Knowledge Graph Embeddings (KGE). Then, we apply Bayesian inference on top of the KG using the embeddings learned during training. Finally, the model can predict lane changes two seconds ahead with 97.95% f1-score, which surpassed the state of the art, and three seconds before changing lanes with 93.60% f1-score.

In the context of autonomous driving, pedestrian behavior prediction is a key component for improving road safety. Presently, many existing prediction models prioritize achieving reliable results, however, they often lack insights into the explainability of each prediction. In this work, we propose a novel approach to pedestrian behavior prediction using knowledge graphs (KG), knowledge graph embeddings (KGE), and a Bayesian Inference process, enabling fully inductive reasoning on KGEs. Our approach aims to consolidate knowledge from annotated datasets through explainable pedestrian features and fuzzy rules, evaluating the importance of these two components within the KG. The entire pipeline has been trained and tested using two datasets: Joint Attention for Autonomous Driving (JAAD) and Pedestrian Situated Intent (PSI). Preliminary results demonstrate the effectiveness of this system in providing explainable clues for pedestrian behavior predictions, even improving results by up to 15% compared to other models. Our approach achieves an F1 score of 0.84 for PSI and 0.82 for JAAD.

Autonomous vehicles (AVs) are being enhanced by introducing wireless communication to improve their intelligence, reliability and efficiency. Despite all of these distinct advantages, the open wireless communication links and connectivity make the AVs’ vulnerability to cyber-attacks. This paper proposes an$L_{2}$-gain-based resilient path following control strategy for AVs under time-constrained denial-of-service (DoS) attacks and external interference. A switching-like path following control model of AVs is first built in the presence of DoS attacks, which is characterized by the lower and upper bounds of the sleeping period and active period of the DoS attacker. Then, the exponential stability and$L_{2}$-gain performance of the resulting switched system are analyzed by using a time-varying Lyapunov function method. On the basis of the obtained analysis results,$L_{2}$-gain-based resilient controllers are designed to achieve an acceptable path-following performance despite the presence of such DoS attacks. Finally, the effectiveness of the proposed$L_{2}$-gain-based resilient path following control method is confirmed by the simulation results obtained for the considered AVs model with different DoS attack parameters.

Global navigation satellite system (GNSS) signals cannot be received indoors, thus to deploy intelligent vehicles in underground car parks other localization methods are needed. In this paper, we use various carpark signs that are widely and uniformly distributed in underground parking lots as localization references. We propose a coarse-to-fine multiscale localization method that relies solely on vision sensors for underground parking lot localization based on a preconstructed lightweight node map. In coarse localization, we propose a semantic keyframe topological localization method to predict the localization range (candidate set of nodes). In node-level localization, we extract features by learning-based neural networks and construct a hybrid k-nearest neighbor (H-KNN) model to search for the closest node within the coarse localization results. In metric localization, we construct plane homography and perspective-n-point (PnP) models, allowing the vehicle’s pose (rotation and translation relative to the closest node) to be computed for refined localization. The proposed method has been tested in two underground parking lots of an office building and a shopping mall with different characteristics. Experimental results demonstrate that root mean square error (RMSE) is 0.38 m and the proposed method exhibits strong robustness in various scenarios.

At present, hydrogen fuel cell electric vehicle (HFCEV) is increasingly affordable to replace petrol vehicles and reduce carbon dioxide emissions. However, the refuelling of the HFCEV is still an essential problem. Specifically, there are not enough hydrogen refuelling stations at hand. In this paper, a hydrogen based microgrid is presented to produce hydrogen to refuel the HFCEV, and different strategies are proposed to guide the HFCEV’s refuelling within the coupled transportation network and power system. First, the HFCEV traffic flow model based on a real-world transportation network is presented. Then, a real-time simulation platform links the Sumo and Matlab is presented. Third, a hydrogen based microgrid to refuel HFCEV is built. Forth, an IEEE 30-node utility grid exporting power model is presented. At last, the real-time hydrogen refuelling of HFCEV through the coupled transportation network and power system is proposed. Four coupled structures are considered, and different HFCEV refuelling strategies (fixed price, dynamic price, LSTM decision price) are compared. The simulation results demonstrate that with the dynamic price, the congestion of the transportation network is improved, the waiting time is reduced by 17.71%, and the time loss of the network is reduced by 13.29%. With reasonable guidance of the price, vehicles choose the selected station to refuel hydrogen and influence the temporal-spatial distribution of the traffic flow of the transportation network. In addition, by adjusting the power station exporting power and the refuelling station importing power, the voltage condition of the power system can be improved.

As an emerging research field, vehicle re-identification (Re-ID) can realize identity search between the vehicles, which plays an important role in the over-the-horizon perception of Vehicle-Infrastructure Cooperative Autonomous Driving (VICAD). At present, due to the lack of data sets, the relevant research on Vehicle-Infrastructure Cooperative (VIC) Re-ID can only be evaluated in the cross-view monitoring test set which leads to the lack of persuasion of the research. Therefore, based on the DAID-V2X dataset of Tsinghua University, this paper constructs a VIC Re-ID dataset “DAIR-V2XReid” from real vehicle scenarios through vehicle-road end target tag association, thereby making it better applicable to the research of VIC Re-ID. Owing to different task scenarios, existing algorithms trained on monitoring test sets are unable to effectively complete the Re-ID task in this new dataset. Therefore, Cross-shot Feature Aggregation Network (CFA-Net) is also proposed in this paper, to tackle the case where a vehicle becomes unrecognizable due to a large change in its visual appearance across different cameras. Firstly, we put forward a camera embedding module and add it to the Backbone, to group different cameras and solve the problem of cross-shot perspective mutation. Secondly, in order to address the situation where background and vehicle division are not distinguishable, we propose a cross-stage feature fusion module, which integrates low-order semantics with high-order semantics. Finally, we use multi-directional attention network to achieve the final feature extraction. The experimental results show that our proposed CFA-Net method achieves new state-of-the-art in DAIR-V2XReid, with mAP of 58.47%.

Abstract This work presents a novel method for predicting vehicle trajectories in highway scenarios using efficient bird’s eye view representations and convolutional neural networks. Vehicle positions, motion histories, road configuration, and vehicle interactions are easily included in the prediction model using basic visual representations. The U-net model has been selected as the prediction kernel to generate future visual representations of the scene using an image-to-image regression approach. A method has been implemented to extract vehicle positions from the generated graphical representations to achieve subpixel resolution. The method has been trained and evaluated using the PREVENTION dataset, an on-board sensor dataset. Different network configurations and scene representations have been evaluated. This study found that U-net with 6 depth levels using a linear terminal layer and a Gaussian representation of the vehicles is the best performing configuration. The use of lane markings was found to produce no improvement in prediction performance. The average prediction error is 0.47 and 0.38 meters and the final prediction error is 0.76 and 0.53 meters for longitudinal and lateral coordinates, respectively, for a predicted trajectory length of 2.0 seconds. The prediction error is up to 50% lower compared to the baseline method.

Real-time and accurate 3-D object detection is one of the foundational technologies for environmental perception in autonomous vehicles. However, the existing second-stage anchor-based 3-D object detection algorithms have high accuracy, but they are challenging in terms of computation complexity and latency. Due to poor perception of spatial features, the accuracy of the existing single-stage anchor-free detection algorithms with low latency are difficult to be implemented into autonomous vehicles. Therefore, we focus on enhancing the spatial perception ability of the anchor-free detection network based on CenterPoints. In this paper, we propose a single-stage anchor-free 3-D object detector CenterPoint-Space-Enhancement (CenterPoint-SE) algorithm and construct an efficient 3-D backbone network to extract fine-grained spatial geometric features by introducing a spatial attention mechanism and residual structure. At the same time, a powerful spatial semantic feature fusion module, the enhancement of feature fusion (EF-Fusion), is designed. In addition, we add a lightweight IoU prediction branch to improve the algorithm’s perception of various object sizes. Finally, we add a foreground point segmentation auxiliary training branch to enable the 3-D backbone to obtain object boundary features. We use the ONCE dataset to train and validate the proposed model, and the results showed that the proposed CenterPoint-SE achieves 70.33 mAP and an inference speed of 17.15 FPS, outperforming other methods.

This paper investigates the control issue of marine surface vehicles (MSVs) subject to internal and external uncertainties without velocity information. Utilizing the specific advantages of adaptive neural network and disturbance observer, a classification reconstruction idea is developed. Based on this idea, a novel adaptive neural-based state observer with disturbance observer is proposed to recover the unmeasurable velocity. Under the vector-backstepping design framework, the classification reconstruction idea and adaptive neural-based state observer are used to resolve the control design issue for MSVs. To improve the control performance, the serial-parallel estimation model is introduced to obtain a prediction error, and then a composite learning law is designed by embedding the prediction error and estimate of lumped disturbance. To reduce the mechanical wear of actuator, a dynamic event triggering protocol is established between the control law and actuator. Finally, a new dynamic event-triggered composite learning adaptive neural output feedback control solution is developed. Employing the Lyapunov stability theory, it is strictly proved that all signals in the closed-loop control system of MSVs are bounded. Simulation and comparison results validate the effectiveness of control solution.

Boresight angle misalignment is a major error source in a mobile LiDAR system (MLS), which directly affects the overall accuracy and quality of MLS scanned point clouds data. However, the current calibration of the boresight angle misalignment mainly relies on artificial target features or a manual adjustment, and the intensive labors dramatically limit the calibration flexibility. To solve these problems, this paper develops a novel on-site calibration method for boresight angle misalignment based on multiple natural features constraints, which can automatically incorporate multiple natural features extracted from surrounding environments to generate more accurate calibration results for MLS boresight angle without used any artificial targets or specific facilities. First of all, an improved 4-points congruent sets (I-4PCS) algorithm is proposed for registering the MLS point clouds in forward and backward scanned overlapping areas and realizing smooth global registration for point clouds data. Secondly, a weight principal component analysis (WPCA) approach is presented to automatically extract the appropriate multiple natural features from the well registered point clouds and establish the appropriate features representation. Thirdly, according to the extracted multiple features, the certain geometric constrains equations for spherical, linear/cylindrical, planar features are established based on a model adjustment strategy. Lastly, the boresight angle misalignment calibration can be achieved through fitting the corresponding geometric constrains equations and minimizing the weighted through a least-squares adjustment process. The experimental results demonstrate that the proposed method can effectively on-site calibrate the boresight angle misalignment error, and the overall performance of MLS is significantly improved after the calibration based on multiple natural features constraints.

Pedestrian motion trajectory prediction is an important task in intelligent driving, and it can provide a valuable reference for the subsequent path decision of intelligent driving. However, so far, there are only a few models in the field of specific pedestrian motion track prediction in intelligent driving from far shot first-person perspective video. To accomplish this task, we proposed a deep learning model for pedestrian motion trajectory prediction from far shot first-person perspective video with four key innovations: a) A macroscopic pedestrian trajectory prediction module is established under the close correlation between neighboring frames to estimate the pedestrian motion track on the whole; b) A relative motion transformation module of vehicle-mounted camera is designed to consider the effect of vehicle-mounted camera’s ego-motion on the pedestrian motion track; c) We set up a circular training module to maintain the number of parameters in our model to simplify and reduce the size of model; d) A new far shot first-person pedestrian motion dataset under intelligent driving is specifically established to train and test the proposed model. The above four modules are integrated into the proposed deep learning model, which achieves state-of-the-art results for predicting pedestrian motion trajectory from both far and close shot first-person perspective video.

This article investigates the event-triggered adaptive fuzzy output feedback setpoint regulation control for the surface vessels. The vessel velocities are noisy and small in the setpoint regulation operation and the thrusters have saturation constraints. A high-gain filter is constructed to obtain the vessel velocity estimations from noisy position and heading. An auxiliary dynamic filter with control deviation as the input is adopted to reduce thruster saturation effects. The adaptive fuzzy logic systems approximate vessel’s uncertain dynamics. The adaptive dynamic surface control is employed to derive the event-triggered adaptive fuzzy setpoint regulation control depending only on noisy position and heading measurements. By the virtue of the event-triggering, the vessel’s thruster acting frequencies are reduced such that the thruster excessive wear is avoided. The computational burden is reduced due to the differentiation avoidance for virtual stabilizing functions required in the traditional backstepping. It is analyzed that the event-triggered adaptive fuzzy setpoint regulation control maintains position and heading at desired points and ensures the closed-loop semi-global stability. Both theoretical analyses and simulations with comparisons validate the effectiveness and the superiority of the control scheme.

The main emphasis of this work is placed on the problem of collaborative coverage path planning for unmanned surface mapping vehicles (USMVs). As a result, the collaborative coverage improved$BA^{*}$algorithm ($C C I B A^{*}$) is proposed. In the algorithm, coverage path planning for a single vehicle is achieved by task decomposition and level map updating. Then a multiple USMV collaborative behavior strategy is designed, which is composed of area division, recall and transfer, area exchange and recognizing obstacles. Moverover, multiple USMV collaborative coverage path planning can be achieved. Consequently, a high-efficiency and high-quality coverage path for USMVs can be implemented. Water area simulation results indicate that our$CCIBA^{*}$brings about a substantial increase in the performances of path length, number of turning, number of units and coverage rate.

In recent years, considerable progress has been made in semantic segmentation of images with favorable environments. However, the environmental perception of autonomous driving under adverse weather conditions is still very challenging. In particular, the low visibility at nighttime greatly affects driving safety. In this paper, we aim to explore image segmentation in low-light scenarios, thereby expanding the application range of autonomous vehicles. The segmentation algorithms for road scenes based on deep learning are highly dependent on the volume of images with pixel-level annotations. Considering the scarcity of labeled large-scale nighttime data, we performed synthetic data collection and data style transfer using images acquired in daytime based on the autonomous driving simulation platform and generative adversarial network, respectively. In addition, we also proposed a novel nighttime segmentation framework (SFNET-N) to effectively recognize objects in dark environments, aiming at the boundary blurring caused by low semantic contrast in low-illumination images. Specifically, the framework comprises a light enhancement network which introduces semantic information for the first time and a segmentation network with strong feature extraction capability. Extensive experiments with Dark Zurich-test and Nighttime Driving-test datasets show the effectiveness of our method compared with existing state-of-the art approaches, with 56.9% and 57.4% mIoU (mean of category-wise intersection-over-union) respectively. Finally, we also performed real-vehicle verification of the proposed models in road scenes of Zhenjiang city with poor lighting. The datasets are available athttps://github.com/pupu-chenyanyan/semantic-segmentation-on-nightime.

In this article, a new event-triggered$H_{\infty }$load frequency control (LFC) approach with dynamic triggered algorithm (DTA) for multi-area nonlinear power systems (NPSs) based on non-fragile proportional integral control (NPI-control) strategy is addressed. Firstly, different from the existing linear single-area LFC model for power systems, an improved nonlinear multi-area model with the performance of large-scale adjustment frequency fluctuation is constructed by considering the phenomenon of overshoots and long-term oscillations. Due to the existence of control uncertainty, it is the first time that the NPI-control scheme is applied to LFC approach for NPSs. Then, the DTA is proposed to adjust the dynamic event-triggered parameters, which reduces the occupation of communication bandwidth and the data computation of NPSs. Furthermore, a modified quadratic form with time-varying matrix and two-side closed functional method are adopted to construct the relaxed Lyapunov-Krasovskii functional, where some slack matrices are unnecessarily positive definite. Based on Lyapunov method, some less-conservatism stability criteria are derived. Utilizing the linear matrix inequality toolbox, the allowable upper bound of time-varying delays and the NPI-controller are obtained. Finally, a numerical example is presented to demonstrate the availability of the approach developed in this work.

This paper investigates the tracking control problem of marine surface vessels (MSVs) in the presence of uncertain dynamics and external disturbances. The facts that actuators are subject to undesirable faults and input saturation are taken into account. Benefiting from the smoothness of the Gaussian error function, a novel saturation function is introduced to replace each nonsmooth actuator saturation nonlinearity. Applying the hand position approach, the original motion dynamics of underactuated MSVs are transformed into a standard integral cascade form so that the vector design method can be used to solve the control problem for underactuated MSVs. By combining the neural network technique and virtual parameter learning algorithm with the vector design method, and introducing an event triggering mechanism, a novel event-triggered indirect neuroadaptive fault-tolerant control scheme is proposed, which has several notable characteristics compared with most existing strategies: 1) it is not only robust and adaptive to uncertain dynamics and external disturbances but is also tolerant to undesirable actuator faults and saturation; 2) it reduces the acting frequency of actuators, thereby decreasing the mechanical wear of the MSV actuators, via the event-triggered control (ETC) technique; 3) it guarantees stable tracking without the aprioriknowledge of the dynamics of the MSVs, external disturbances or actuator faults; and 4) it only involves two parameter adaptations—a virtual parameter and a lower bound on the uncertain gains of the actuators—and is thus more affordable to implement. On the basis of the Lyapunov theorem, it is verified that all signals in the tracking control system of the underactuated MSVs are bounded. Finally, the effectiveness of the proposed control scheme is demonstrated by simulations and comparative results.

Understanding complex scenarios from in-vehicle cameras is essential for safely operating autonomous driving systems in densely populated areas. Among these, intersection areas are one of the most critical as they concentrate a considerable number of traffic accidents and fatalities. Detecting and understanding the scene configuration of these usually crowded areas is then of extreme importance for both autonomous vehicles and modern Advanced Driver Assistance Systems (ADAS), aimed at preventing road crashes and increasing the safety of Vulnerable Road Users (VRU). This work investigates how to classify intersection areas from RGB images using well-consolidated neural network approaches along with a method to enhance the results based on the teacher/student training paradigm. An extensive experimental activity aimed at identifying the best input configuration and evaluating different network parameters on both the well-known KITTI dataset and the new KITTI-360 sequences shows that our method outperforms current state-of-the-art intersection classification approaches on per-frame basis, proving the effectiveness of the proposed learning scheme.

Autonomous vehicles navigate in dynamically changing environments under a wide variety of conditions, being continuously influenced by surrounding objects. Mod - elling interactions among agents is essential for accurately forecasting other agents' behaviour and achieving safe and comfortable motion planning. In this work, we propose SCOUT, a novel Attention-based Graph Neural Network that uses a flexible and generic representation of the scene as a graph for modelling interactions, and predicts socially - consistent trajec - tories of vehicles and Vulnerable Road Users (VRU s) under mixed traffic conditions. We explore three different attention mechanisms and test our scheme with both bird-eye - view and on-vehicle urban data, achieving superior performance than existing state-of - the-art approaches on InD and ApolloScape Trajectory benchmarks. Additionally, we evaluate our model's flexibility and transferability by testing it under completely new scenarios on RounD dataset. The importance and influence of each interaction in the final prediction is explored by means of Integrated Gradients technique and the visualization of the attention learned.

Understanding the behavior of road users is of vital importance for the development of trajectory prediction systems. In this context, the latest advances have focused on recurrent structures, establishing the social interaction between the agents involved in the scene. More recently, simpler structures have also been introduced for predicting pedestrian trajectories, based on Transformer Networks, and using positional information [1]. They allow the individual modelling of each agent's trajectory separately without any complex interaction terms. Our model exploits these simple structures by adding augmented data (position and heading), and adapting their use to the problem of vehicle trajectory prediction in urban scenarios in prediction horizons up to 5 seconds. In addition, a cross-performance analysis is performed between different types of scenarios, including highways, intersections and roundabouts, using recent datasets (inD, rounD, highD and INTERACTION). Our model achieves state-of-the-art results and proves to be flexible and adaptable to different types of urban contexts.

A multimode hybrid automaton is proposed for setting vehicle platoon modes with velocity, distance, length, lane position, and other state information. Based on a vehicle platoon shift movement under different modes, decisions are made based on key conditional actions, such as sudden acceleration changes because of vehicle distance changes, emergency braking to avoid collisions and free-lane changing choices adapted to various traffic conditions, so as to ensure effortless movement and safety in the multimode shift. With a 3-degree (longitudinal, lateral, and yaw directions) of the freedom coupled model, a hybrid vehicle platoon controller is proposed using nonsingular terminal sliding-mode control to ensure fast and steady tracking on the hybrid automaton outputs during the multimode shift process. The convergence of the hybrid controller in finite time is also analyzed with the Lyapunov exponential stability. The analysis result proves that the proposed controller not only ensures the stability of the individual vehicle and the vehicle platoon but also ensures the stability of the multimode shift movement system. The proposed cooperative driving strategy for vehicle platoon is evaluated using simulations, where varying traffic conditions and the influence of cutting off are considered in conjunction with demonstration simulations of a vehicle platoon's cruising, following, lane changing, overtaking, and moving in/out of garage functions.

Accurate positioning is an essential requirement of autonomous vehicular navigation system (AVNS) for safe driving. Although the vehicle position can be obtained in global position system friendly environments, in GPS denied environments (such as suburb, tunnel, forest, or underground scenarios) the positioning accuracy of AVNS is easily reduced by the trajectory error of the vehicle. In order to solve this problem, the plane, sphere, cylinder and cone are often selected as the ground control targets to eliminate the trajectory error for AVNS. However, these targets usually suffer from the limitations of incidence angle, measuring range, scanning resolution, and point cloud density, etc. To bridge this research gap, an adaptive continuum shape constraint analysis (ACSCA) method is presented in this article to design a new target with optimized identifiable specific shape to eliminate the trajectory error for AVNS. First of all, according to the proposed ACSCA method, we conduct extensive numerical simulations to explore the optimal ranges of the vertexes and the faces for target shape design, and based on these trials, the optimal target shape is found as icosahedron, which composes of ten vertexes, 20 faces and combines the properties of plane and volume target. Moreover, the algorithm of automatic detection and coordinate calculation is developed to recognize the icosahedron target and calculate its coordinates information for AVNS. Finally, a series of experimental investigation were performed to evaluate the effectiveness of the designed icosahedron target in GPS denied environments. The experimental results demonstrate that compared with the plane, sphere, cylinder and cone targets, the developed icosahedron target can produce better performances than the above targets in terms of the clustered minimum registration error, ambiguity and range of field-of-view; also can significantly improve the positioning accuracy of AVNS in GPS denied environments.

This paper addresses the co-design problem of a fault detection filter and controller for a networked-based unmanned surface vehicle (USV) system subject to communication delays, external disturbance, faults, and aperiodic denial-of-service (DoS) jamming attacks. First, an event-triggering communication scheme is proposed to enhance the efficiency of network resource utilization while counteracting the impact of aperiodic DoS attacks on the USV control system performance. Second, an event-based switched USV control system is presented to account for the simultaneous presence of communication delays, disturbance, faults, and DoS jamming attacks. Third, by using the piecewise Lyapunov functional (PLF) approach, criteria for exponential stability analysis and co-design of a desired observer-based fault detection filter and an event-triggered controller are derived and expressed in terms of linear matrix inequalities (LMIs). Finally, the simulation results verify the effectiveness of the proposed co-design method. The results show that this method not only ensures the safe and stable operation of the USV but also reduces the amount of data transmissions.

We present an innovative personal rapid transit technology compatible with current metro infrastructures, named OPTIMOTUS. The key of this technology is that passengers can travel without stops, thus multiplying several times the effective travel speed. In metro lines with shorter separation between stations (5 stations), passengers travel speed could be up to 3.5 times faster with OPTIMOTUS than with conventional metro trains. In addition, its implementation costs would be relatively low since the vehicles are designed to be compatible with current metro railways. However, there are still many open issues to be engineered before claiming the full viability of the technology.

Accurate localization is a crucial step for intelligent vehicles (IVs). And vision-based localization methods are promising due to its good accuracy and low cost. However, vision-based methods are usually not robust enough due to the errors of matching similar road scenarios. In this paper, we proposed a visual map-based localization method, called multi-view site matching (MVSM). We proposed using two camera views (i.e., downward-view and front-view) to construct visual map. The visual map consists of a serial of nodes. Each node encodes the features of the road, the 2D structure, and the poses of the vehicle. Based on the constructed visual map, we proposed a multi-scale method for accurate vehicle localization. In coarse localization, we adopt a topological model to obtain a set of candidate nodes from visual map. Furthermore, holistic features from front view are matched within the candidates such that the best matched node is determined for image-level localization. In metric localization, the best matched is first verified with the local features from downward view. And the vehicle pose is finally computed by utilizing the 2D structure from the verified nodes in the map. In the experiment, the proposed MVSM method has been tested with actual field data covering different pavement types in different seasons. The proposed MVSM method can achieve less than 0.20m mean localization errors. Compared to existing vision-based methods, the proposed method utilizes two views to enhance image-level localization and 2D pavement structure to improve metric localization so as to greatly improve the overall localization performance.

This paper aims to solve the path following problem for an underactuated unmanned-surface-vessel (USV) based on deep reinforcement learning (DRL). A smoothly-convergent DRL (SCDRL) method is proposed based on the deep Q network (DQN) and reinforcement learning. In this new method, an improved DQN structure was developed as a decision-making network to reduce the complexity of the control law for the path following of a three-degree of freedom USV model. An exploring function was proposed based on the adaptive gradient descent to extract the training knowledge for the DQN from the empirical data. In addition, a new reward function was designed to evaluate the output decisions of the DQN, and hence, to reinforce the decision-making network in controlling the USV path following. Numerical simulations were conducted to evaluate the performance of the proposed method. The analysis results demonstrate that the proposed SCDRL converges more smoothly than the traditional deep Q learning while the path following error of the SCDRL is comparable to existing methods. Thanks to good usability and generality of the proposed method for USV path following, it can be applied to practical applications.

Road detection and segmentation is a crucial task in computer vision for safe autonomous driving. With this in mind a new net architecture (3D-DEEP) and its end-to-end training methodology for CNN-based semantic segmentation is described along this paper for. The method relies on disparity filtered and LiDAR projected images for three-dimensional information and image feature extraction through fully convolutional networks architectures. The developed models were trained and validated over Cityscapes dataset using just fine annotation examples with 19 different training classes, and over KITTI road dataset. 72.32% mean intersection over union (mIoU) has been obtained for the 19 Cityscapes training classes using the validation images. On the other hand, over KITTI dataset the model has achieved an F1 error value of 97.85% in validation and 96.02% using the test images.

Pedestrian crossing prediction is a crucial task for autonomous driving. Numerous studies show that an early estimation of the pedestrian's intention can decrease or even avoid a high percentage of accidents. In this paper, different variations of a deep learning system are proposed to attempt to solve this problem. The proposed models are composed of two parts: a CNN-based feature extractor and an RNN module. All the models were trained and tested on the JAAD dataset. The results obtained indicate that the choice of the features extraction method, the inclusion of additional variables such as pedestrian gaze direction and discrete orientation, and the chosen RNN type have a significant impact on the final performance.

While driving on highways, every driver tries to be aware of the behavior of surrounding vehicles, including possible emergency braking, evasive maneuvers trying to avoid obstacles, unexpected lane changes, or other emergencies that could lead to an accident. In this paper, human's ability to predict lane changes in highway scenarios is analyzed through the use of video sequences extracted from the PREVENTION dataset, a database focused on the development of research on vehicle intention and trajectory prediction. Thus, users had to indicate the moment at which they considered that a lane change maneuver was taking place in a target vehicle, subsequently indicating its direction: left or right. The results retrieved have been carefully analyzed and compared to ground truth labels, evaluating statistical models to understand whether humans can actually predict. The study has revealed that most participants are unable to anticipate lane-change maneuvers, detecting them after they have started. These results might serve as a baseline for AI's prediction ability evaluation, grading if those systems can outperform human skills by analyzing hidden cues that seem unnoticed, improving the detection time, and even anticipating maneuvers in some cases.

With the aid of global position system (GPS), mobile laser scanning (MLS) is able to provide 3-D geo-referenced point cloud that has centimeter-level accuracy. The MLS accuracy, however, degrades significantly due to the trajectory errors of the laser scanner and the residual systematic errors from the geo-referencing transformation process in the GPS-free environments. To solve this problem, this article presents a novel integration algorithm based on the weighted total least squares (WTLS) and the 3-D conformal coordinate transformation (3DCCT). In this new method, the 3-D point measurement model and the error propagation parameter vector in the MLS can be updated in real-time, and they can also adjust the geo-referenced coordinate transformation parameters and eliminate the influences of the residual systematic errors during MLS. In this article, the MLS mathematical model is first established, followed up by a detailed analysis for MLS error budget interpreting the effects of the individual error sources. Second, WTLS is used to correct the 3-D point measurement model of MLS and the error of propagation parameter vector; 3DCCT, WTLS, and ground control target feature constraints are applied to eliminate the residual systematic errors in the geo-referencing transformation process. Finally, several data sets from outdoor scenarios are used to evaluate and validate the proposed method. The experimental results demonstrate that the proposed method can significantly improve the overall accuracy of the MLS system.

The accuracy of geometric parameters (mainly referred to the incidence angle and measuring distance) in a terrestrial laser scanner (TLS) is not only influenced by the TLS intrinsic systematic instrumental error but also the extrinsic received intensity data. However, the current error compensation methods for geometric parameters mainly focus on the calibration of TLS intrinsic systematic instrumental error and rarely consider the extrinsic intensity data correction. For this reason, this article presents a new method integrating the TLS intrinsic systematic instrumental error calibration and extrinsic intensity data correction to compensate the TLS geometric parameter error. The error compensation procedure is implemented as follows. First, the error compensation mathematical model integrated with TLS intrinsic systematic instrumental error calibration parameters and extrinsic intensity data correction coefficient is established. Second, the hybrid harmonic analysis (HA) and the adaptive wavelet neural network (AWNN) algorithm are proposed to calculate the TLS incidence angle error compensation values. Subsequently, the cubic spline interpolation (CSI) is applied to compute the measuring distance error compensate values. Finally, the TLS (model FARO Focus S150) and the hemispherical angle calibration instrument were used to evaluate the proposed compensation method. The experimental results demonstrate that the geometric parameters are significantly influenced by the intensity data received from TLS, and the proposed method can effectively improve the overall accuracy of the TLS incidence angle and measuring distance.

In the last century, the automotive industry has arguably transformed society, being one of the most complex, sophisticated, and technologically advanced industries. Autonomous vehicles (AVs) are a main concept in the future of Intelligent Transportation Systems (ITS) since they provide an increase in safety and road efficiency. One of the most critical aspects of managing AVs is their behavior in proximity of intersections. Several research centers are developing algorithms to solve the intersections management, trying to avoid collisions and traffic congestion. As well as, given that many of these interactions transmit sensitive data such as identification, position, and speed of the vehicle, a high level of security and privacy insurance is a prerequisite for broad acceptation of these communication systems. In this paper, in order to address the issues those issues we propose a system that combines blockchain technology effectively to support the communication and the transaction between vehicles. The combination between FRFP and blockchain allows us to verify if all the AVs have the same ledger version (e.g the same priority list) to cross the intersection without collisions; as well as, in case of inconsistencies to establish an emergency situation to avoid any collision.

This paper proposes a variable structure control approach for vehicles platooning based on a hierarchical fuzzy logic. The leader-follower vehicle dynamics with model uncertainties is discussed from the viewpoint of a consensus problem. A practical two-layer fuzzy control for the platooning is designed by employing two common spacing policies to ensure system robustness in different scenarios. The two policies, i.e., constant distance and constant time headway, utilize the predecessor-successor information flow from the immediate predecessor and follower other than controlled vehicles. The first layer of the fuzzy system combines spacing control with velocity-acceleration control to achieve a rapid tracking for the desired control commands, and the second layer combines the sliding mode control to adaptively compensate for reducing the state errors caused by parameter uncertainties and disturbances. Shift between different controller parameters is based on performance boundaries to guarantee the stability of individual vehicle and platooning for arbitrary initial spacing and velocity errors. These performance boundaries can be determined by using a Lyapunov method with exponential stability. Simulation of a ten-vehicle large platooning with two spacing policies shows that the control performance of the newly proposed method is effective and promising.

According to several reports published by worldwide organizations, thousands of pedestrians die in road accidents every year. Due to this fact, vehicular technologies have been evolving with the intent of reducing these fatalities. This evolution has not finished yet, since, for instance, the predictions of pedestrian paths could improve the current automatic emergency braking systems. For this reason, this paper proposes a method to predict future pedestrian paths, poses, and intentions up to 1 s in advance. This method is based on balanced Gaussian process dynamical models (B-GPDMs), which reduce the 3-D time-related information extracted from key points or joints placed along pedestrian bodies into low-dimensional spaces. The B-GPDM is also capable of inferring future latent positions and reconstruct their associated observations. However, learning a generic model for all kinds of pedestrian activities normally provides less accurate predictions. For this reason, the proposed method obtains multiple models of four types of activity, i.e., walking, stopping, starting, and standing, and selects the most similar model to estimate future pedestrian states. This method detects starting activities 125 ms after the gait initiation with an accuracy of 80% and recognizes stopping intentions 58.33 ms before the event with an accuracy of 70%. Concerning the path prediction, the mean error for stopping activities at a time-to-event (TTE) of 1 s is 238.01 ± 206.93 mm and, for starting actions, the mean error at a TTE of 0 s is 331.93 ± 254.73 mm.

In recent years, autonomous driving is becoming a very hot topic for both researchers and car manufacturers. Indeed, around the world new discoveries have been published. In this work we present an innovative Osmotic Computing solution for self adapting city traffic in autonomous vehicle environment. The Vehicular-to-Vehicular (V2V) and Vehicular to Edge-Cloud (V2EC) interactions inside specific areas of the City are considered: the interconnections. The Framework we are creating is able to adapt on a Dynamic Environment where Vehicles, Pedestrians and Physical Infrastructures can interact each other, offering continuous information on interconnections status and city traffic in general.

Many research works have contributed to achieve SAE levels 3 and 4 in some pre-defined areas under certain restrictions. A deeper scene understanding and precise predictions of drivers intentions are needed to continue improving autonomous driving capabilities to reach higher SAE levels. Deployment of accurate and detailed datasets could be considered as one of the most pressing needs to enhance autonomous driving capabilities. This work presents a novel data acquisition methodology for on-road vehicle trajectory collection. The proposed sensor setup improves the range and detection accuracy by using a high accuracy laser scanner covering 360° and two high-speed and high-resolution cameras. The sensor fusion increases the labelling resolution and extends the detection range sporting the best of each sensor. A Median Flow tracking algorithm and a Convolutional Neural Network enable a semi-automatic labelling process, which reduces the effort to create detailed annotated datasets. High accurate trajectories are reconstructed with few manual annotations up to 60m with a mean error below 2 cm. This methodology has been developed with a view to creating a dataset which enables the development of advanced vehicle trajectory prediction systems, and thus to contribute to human-like automated driving.

Welcome to the third issue of the IEEE Transactions on Computational Social Systems (TCSS) for 2018.

The second edition of the grand cooperative driving challenge (GCDC2016) was held in The Netherlands in May 2016. Ten international teams participated in the two competition scenarios designed for GCDC2016: platoon merging and intersection. This paper describes the design and development of DRIVERTIVE, a DRIVERless cooperaTIve VEhicle, which aims to advance cooperative automation. The purpose of this paper is to give a general overview of the different designs used to adapt a factory vehicle, with no access to low-level control systems, into a fully-automated cooperative vehicle fit to compete in GCDC2016. The approach taken was pragmatic: different pre-existing techniques for control, state estimation, data fusion, communication, and data degradation were combined and experimentally validated in real-world scenarios, together with other vehicles with different implementations. Our main conclusion is that cooperative autonomous driving is feasible among very different implementations of the communication protocols and using completely different autonomous vehicles.

In this paper the performance of ITS-G5A communications for an autonomous driving application is analyzed in a real high-density scenario. The data was collected during the cooperative platooning tests that took place in Helmond in the frame of the Grand Cooperative Driving Challenge 2016. In the competition, between 8-10 autonomous vehicles formed two platoons in different lanes and were required to merge into a predefined competition zone. The performance is characterized using CAM CCDFs which serves as a base for the evaluation of a Cooperative Adaptive Cruise Control application. Two important effects has been identified that affect to the reliability of the communications. Firstly, there is a degradation with the distance that appears to be stronger for cars and more gentle for trucks. This indicates that occlusions heavily affect the connectivity of ITS-G5A. Secondly, the reliability is below expectations and some of the vehicles perform consistently worse than others. Although further investigation is required, a possible explanation for this is that a highly congested channel is making some of the vehicles get stuck and are not able to regularly access the channel.

This paper addresses the problem of curb detection for ADAS or autonomous navigation in urban scenarios. The algorithm is based on clouds of 3D points. It is evaluated using 3D information from a pair of stereo cameras and a LIDAR. Curbs are detected based on road surface curvature. The curvature estimation requires a dense point cloud, therefore the density of the LIDAR cloud has been augmented using Iterative Closest Point (ICP) based on the previous scans. The proposed algorithm can deal with curbs of different curvature and heights, from as low as 3 cm, in a range up to 20 m (whenever that curbs are connected in the curvature image). The curb parameters are modeled using straight lines and compared to the ground-truth using the lateral error as the key parameter indicator. The ground-truth sequences were manually labeled on urban images from the KITTI dataset and made publicly available for the scientific community.

Abstract: In this paper, a stereo- and infrastructure-based pedestrian detection system is presented to deal with infrastructure-based pedestrian safety measurements as well as to assess pedestrian behaviour modelling meth-ods. Pedestrian detection is performed by region growing over temporal 3D density maps, which are obtained by means of stereo reconstruction and background modelling. 3D tracking allows to correlate the pedestrian position with the different pedestrian crossing regions (waiting and crossing areas). As an example of an infrastructure safety system, a blinking luminous traffic sign is switched on to warn the drivers about the pres-ence of pedestrians in the waiting and the crossing regions. The detection system provides accurate results even for nighttime conditions: an overall detection rate of 97.43 % with one false alarm per each 10 minutes. In addition, the proposed approach is validated for being used in pedestrian behaviour modelling, applying logistic regression to model the probability of a pedestrian to cross or wait. Some of the predictor variables are automatically obtained by using the pedestrian detection system. Other variables are still needed to be labelled using manual supervision. A sequential feature selection method showed that time-to-collision and pedestrian waiting time (both variables automatically collected) are the most significant parameters when predicting the pedestrian intent. An overall predictive accuracy of 93.10 % is obtained, which clearly validates the proposed methodology. 1

Driver Assistance Systems have achieved a high level of maturity in the latest years. As an example of that, sophisticated pedestrian protection systems are already available in a number of commercial vehicles from several OEMs. However, accurate pedestrian path prediction is needed in order to go a step further in terms of safety and reliability, since it can make the difference between effective and non-effective intervention. In this paper, we consider the three-dimensional pedestrian body language in order to perform path prediction in a probabilistic framework. For this purpose, the different body parts and joints are detected using stereo vision. We propose the use of GPDM (Gaussian Process Dynamical Models) for reducing the high dimensionality of the input feature vector (composed by joints and displacement vectors) in the 3D pose space and for learning the pedestrian dynamics in a latent space. Experimental results show that accurate path prediction can be achieved at a time horizon of ≈ 0.8 s.

Backing-out maneuvers in perpendicular or angle parking lots are one of the most dangerous maneuvers, specially in cases where side parked cars block the driver view of the potential traffic flow. In this paper a new vision-based Advanced Driver Assistance System (ADAS) is proposed to automatically warn the driver in such scenarios. A monocular gray-scale camera is installed at the back-right side of the vehicle. A Finite State Machine (FSM) defined according to three CAN-Bus variables and a manual signal provided by the user is used to handle the activation/deactivation of the detection module. The proposed oncoming traffic detection module computes spatiotemporal images from a set of pre-defined scan-lines which are related to the position of the road. A novel spatio-temporal motion descriptor is proposed (STHOL) accounting the number of lines, their orientation and length of the spatio-temporal images. A Bayesian framework is used to trigger the warning signal using multivariate normal density functions. Experiments are conducted on image data captured from a vehicle parked at different locations of an urban environment, including different lighting conditions. We demonstrate that the proposed approach provides robust results maintaining processing rates close to real-time.

In this paper, a real-time free space detection system is presented using a medium-cost lidar sensor and a low cost camera. The extrinsic relationship between both sensors is obtained after an off-line calibration process. The lidar provides measurements corresponding to 4 horizontal layers with a vertical resolution of 3.2 degrees. These measurements are integrated in time according to the relative motion of the vehicle between consecutive laser scans. A special case is considered here for Spanish speed humps, since these are usually detected as an obstacle. In Spain, speed humps are directly related with raised zebra-crossings so they should have painted white stripes on them. Accordingly the conditions required to detect a speed hump are: detect a slope shape on the road and detect a zebra crossing at the same time. The first condition is evaluated using lidar sensor and the second one using the camera.

Over the past few years, advanced driver-assistance systems (ADASs) have become a key element in the research and development of intelligent transportation systems (ITSs) and particularly of intelligent vehicles. Many of these systems require accurate global localization information, which has been traditionally performed by the Global Positioning System (GPS), despite its well-known failings, particularly in urban environments. Different solutions have been attempted to bridge the gaps of GPS positioning errors, but they usually require additional expensive sensors. Vision-based algorithms have proved to be capable of tracking the position of a vehicle over long distances using only a sequence of images as input and with no prior knowledge of the environment. This paper describes a full solution to the estimation of the global position of a vehicle in a digital road map by means of visual information alone. Our solution is based on a stereo platform used to estimate the motion trajectory of the ego vehicle and a map-matching algorithm, which will correct the cumulative errors of the vision-based motion information and estimate the global position of the vehicle in a digital road map. We demonstrate our system in large-scale urban experiments reaching high accuracy in the estimation of the global position and allowing for longer GPS blackouts due to both the high accuracy of our visual odometry estimation and the correction of the cumulative error of the map-matching algorithm. Typically, challenging situations in urban environments such as nonstatic objects or illumination exceeding the dynamic range of the cameras are shown and discussed.

This paper presents the results of a set of extensive experiments carried out under both daytime and nighttime real traffic conditions. The data were captured using an enhanced or extended Floating Car Data system (xFCD) that includes a stereo vision sensor for detecting the local traffic ahead. The collected information is then used to propose a novel approach to the level-of-service (LOS) calculation. This calculation uses information from both the xFCD and the magnetic loops deployed in the infrastructure to construct a speed/occupancy hybrid plane that characterizes the traffic state of a continuous route. In the xFCD system, the detection component implies the use of previously developed monocular approaches in combination with new stereo vision algorithms that add robustness to the detection and increase the accuracy of the measurements corresponding to relative distance and speed. In addition to the stereo pair of cameras, the vehicle is equipped with a low-cost Global Positioning System (GPS) and an electronic device for controller-area-network bus interfacing. The xFCD system has been tested in a 198-min sequence recorded in real traffic scenarios under different weather and illumination conditions. The results are promising and demonstrate that the xFCD system is ready for being used as a source of traffic status information. As an indicative example of the developed xFCD system, we construct a novel route LOS calculation that combines hybrid information about speed and occupancy from both the xFCD system and the magnetic loops in the infrastructure.

The ten papers in this special issue cover the full range of cooperative technologies in Intelligent Transportation Systems, from V2V and V2I, including cooperative traffic management to vehicle-to-driver cooperation. These papers are summarized here.

This paper presents the results of a set of extensive experiments carried out in daytime and nighttime conditions in real traffic using an enhanced or extended Floating Car Data system (xFCD) that includes a stereo vision sensor for detecting the local traffic ahead. The detection component implies the use of previously monocular approaches developed by our group in combination with new stereo vision algorithms that add robustness to the detection and increase the accuracy of the measurements corresponding to relative distance and speed. Besides the stereo pair of cameras, the vehicle is equipped with a low-cost GPS and an electronic device for CAN Bus interfacing. The xFCD system has been tested in a 198-minutes sequence recorded in real traffic scenarios with different weather and illumination conditions, which represents the main contribution of this paper. The results are promising and demonstrate that the system is ready for being used as a source of traffic state information.

Computer vision techniques applied to systems used on road maintenance, which are related either to traffic signs or to the road itself, are playing a major role in many countries because of the higher investment on public works of this kind. These systems are able to collect a wide range of information automatically and quickly, with the aim of improving road safety. In this context, the correct visibility of traffic signs and panels is vital for the safety of drivers. This paper describes an approach to the VISUAL Inspection of Signs and panEls (“VISUALISE”), which is an automatic inspection system, mounted onboard a vehicle, which performs inspection tasks at conventional driving speeds. VISUALISE allows for an improvement in the awareness of the road signaling state, supporting planning and decision making on the administration's and infrastructure operators' side. A description of the main computer vision techniques and some experimental results obtained from thousands of kilometers are presented. Finally, the conclusions of the system are described.

Collision avoidance is one of the most difficult and challenging automatic driving operations in the domain of intelligent vehicles. In emergency situations, human drivers are more likely to brake than to steer, although the optimal maneuver would, more frequently, be steering alone. This statement suggests the use of automatic steering as a promising solution to avoid accidents in the future. The objective of this paper is to provide a collision avoidance system (CAS) for autonomous vehicles, focusing on pedestrian collision avoidance. The detection component involves a stereo-vision-based pedestrian detection system that provides suitable measurements of the time to collision. The collision avoidance maneuver is performed using fuzzy controllers for the actuators that mimic human behavior and reactions, along with a high-precision Global Positioning System (GPS), which provides the information needed for the autonomous navigation. The proposed system is evaluated in two steps. First, drivers' behavior and sensor accuracy are studied in experiments carried out by manual driving. This study will be used to define the parameters of the second step, in which automatic pedestrian collision avoidance is carried out at speeds of up to 30 km/h. The performed field tests provided encouraging results and proved the viability of the proposed approach.

This paper describes an improved stereovision system for the anticipated detection of car-to-pedestrian accidents. An improvement of the previous versions of the pedestrian-detection system is achieved by compensation of the camera's pitch angle, since it results in higher accuracy in the location of the ground plane and more accurate depth measurements. The system has been mounted on two different prototype cars, and several real collision-avoidance and collision-mitigation experiments have been carried out in private circuits using actors and dummies, which represents one of the main contributions of this paper. Collision avoidance is carried out by means of deceleration strategies whenever the accident is avoidable. Likewise, collision mitigation is accomplished by triggering an active hood system.

This work is driven by our vision that cybernetic transport systems (CTSs) based on fully automated urban vehicles (cybercars) will be seen on city roads and new dedicated infrastructures in the near future. These automated vehicles can be heterogeneous systems, such as human-driven traffic, each having different features and functionalities. In our case, we have developed a control architecture that can manage automatic driving of two cars: 1) CyCabs and 2) automated mass-produced cars. This architecture is interoperable and generates humanlike control of vehicles in any situation. Installation and communication with each vehicle are easy. The autonomous route-tracking behaviors are similar, even if the mechanical, electronic, software, and hardware configurations are different for both cars. The results of the developments shown in this paper are part of the European Union (EU) CyberCars-2 Project, which is currently under deployment.

In this paper a ring of calibrated and synchronized cameras is used for achieving robot and obstacle localization inside a common observed area. To avoid complex appearance matching derived from the wide-baseline arrangement of cameras, a metric occupancy grid is obtained by intersection of silhouettes projected onto the floor. A particle filter is proposed for tracking multiple objects by using the grid as observation data. A clustering algorithm is included in the filter to increase the robustness and adaptability of the multimodal estimation task. To preserve identity of the robot from the set of tracked objects, odometry readings are used to compute a maximum likelihood (ML) global trajectory identification. As a proof of concept, real results are obtained in a long sequence with a mobile robot moving in a human-cluttered scene.

This paper describes a comprehensive combination of feature extraction methods for vision-based pedestrian detection in Intelligent Transportation Systems. The basic components of pedestrians are first located in the image and then combined with a support-vector-machine-based classifier. This poses the problem of pedestrian detection in real cluttered road images. Candidate pedestrians are located using a subtractive clustering attention mechanism based on stereo vision. A components-based learning approach is proposed in order to better deal with pedestrian variability, illumination conditions, partial occlusions, and rotations. Extensive comparisons have been carried out using different feature extraction methods as a key to image understanding in real traffic conditions. A database containing thousands of pedestrian samples extracted from real traffic images has been created for learning purposes at either daytime or nighttime. The results achieved to date show interesting conclusions that suggest a combination of feature extraction methods as an essential clue for enhanced detection performance

The framework of this paper is the robotics navigation inside buildings using WiFi signal strength measure. This navigation is achieved using a partially observable Markov decision process (POMDP). In the localization phase we used WiFi signal strength and ultrasound measures as observations. The localization system works in two stages: map construction and localization stage. The map construction stage usually requires a great effort, therefore in this paper we address the problem of minimizing this calibration effort using an automatic training method. We describe the method based on simultaneous localization and mapping (SLAM) techniques and in a robust local navigation task. This automatic method is compared with a manual method to obtain a deterministic map. Also we demonstrate that using this one in a on-line training stage the system is able to adapt the WiFi map to the variations of the WiFi signal measure. Additionally, we analyze the optimal parameters for this automatic training system. The system has been tested in a real environment using two commercial robotic platforms. Some experimental results and the conclusions are presented

This paper presents a nonintrusive prototype computer vision system for monitoring a driver's vigilance in real time. It is based on a hardware system for the real-time acquisition of a driver's images using an active IR illuminator and the software implementation for monitoring some visual behaviors that characterize a driver's level of vigilance. Six parameters are calculated: Percent eye closure (PERCLOS), eye closure duration, blink frequency, nodding frequency, face position, and fixed gaze. These parameters are combined using a fuzzy classifier to infer the level of inattentiveness of the driver. The use of multiple visual parameters and the fusion of these parameters yield a more robust and accurate inattention characterization than by using a single parameter. The system has been tested with different sequences recorded in night and day driving conditions in a motorway and with different users. Some experimental results and conclusions about the performance of the system are presented

The work described in this paper explores a new solution for tracking multiple and dynamic objects in complex environments. An XPF (extended particle filter) is used to implement a multimodal distribution that represents the most probable estimation for each object position and velocity. A standard PF (particle filter) cannot be used with a variable number of obstacles; some other solutions have been tested in different previous works, but most of them require heavy computational resources at least for a high number of obstacles to be tracked. The solution described here includes a clustering procedure that increases the robustness of the probabilistic process in order to provide on-line adaptation to the variable number of clusters. The result is the XPFCP: extended particle filter with clustering process. The presented algorithm has been tested using stereovision measurements; the results included in the paper show the efficiency of the proposed system.

This paper presents a robot navigation system for indoor environments using a partially observable Markov decision process (POMDP) based on WiFi signal strength and ultrasound observations. The paper represents the first one in using WiFi sensor readings as an observation in a POMDP. We present an algorithm based on an EM-SLAM that we called WSLAM (Wifi simultaneous localization and mapping) that is able to learn the observation and transition matrix in autonomous mode. With this algorithm we obtain a minimum calibration effort. We demonstrate that this system is useful to navigate in indoor environments with a real robot. Some experimental results are shown. Finally, the conclusions and future works are presented.

There is a broad range of robotics technologies that are currently being applied to the generic topic of intelligent transportation systems (ITS). One of the most important research topics in this field is adaptive cruise control (ACC), aiming at adapting the vehicle speed to a predefined value while keeping a safe gap with regard to potential obstacles. For this purpose, a monocular vision system provides the distance between the ego vehicle and the preceding vehicle on the road. The complete system can be understood as a vision-based ACC controller, based on fuzzy logic, which assists the velocity vehicle control offering driving strategies and actuation over the throttle of a car. This controller is embedded in an automatic driving system installed in two testbed mass-produced cars operating in a real environment. The results obtained in these experiments show a very good performance of the vision-based gap controller, which is adaptable to all speeds and safe gap selections.

This work presents an intelligent transportation system (ITS) that was implemented on an autonomous vehicle designed to perform global navigation missions on a network of unmarked roads. This is the first step toward the complete implementation of ITS in urban environments, which is the long-term goal of this work. Using a global positioning system, global navigation is achieved by means of a global planner and a task manager that recurrently coordinate the execution of vision-based perception tasks for the road tracking of nonstructured roads and the navigation of intersections. In addition, a vision-based vehicle-detection task has been developed, which endows the global navigation system with a reactive capacity. The complete system has been tested on the BABIECA prototype vehicle, which was autonomously driven for hundreds of kilometers around a private circuit, designed to emulate an urban quarter, at speeds of up to 50 km/h, successfully carrying out different navigation missions. During the tests, the vehicle drove itself across crossroads and performed the appropriate turning maneuvers at intersections. It also demonstrated its robustness with regard to shadows, road texture, weather conditions, and changing illumination.

In this paper a system for off-line traffic sign detection is shown. Matlab-image-processing toolbox is used for this purpose. The vision-based traffic sign detection module developed in this work manages 172/spl times/352 color images in RGB (red, green, blue) format. The first step in the algorithm is to obtain the gradient image and its vertical edge projection. In a second step, a color and shape analysis is performed.

This paper presents the development of a probabilistic algorithm based on an Extended Kalman Filter (EKF), used to estimate the absolute position of an indoor autonomous robot. With EKF it is possible to fuse relative and absolute positioning data, including some kind of uncertainty related to sensory systems. To reach this objective it is necessary to do an important model analysis to enable the on-line adaptation of the estimation algorithm. The development presented in this paper has been designed for an autonomous wheelchair, whose real-time and reliability constraints have to be taken into account in the algorithm.