Current autonomous driving technology typically relies on high-precision (HD) maps to ensure safe, reliable, and accurate navigation in urban environments. While these maps provide essential road information, their creation and maintenance are costly, limiting their widespread application. To mitigate this reliance, we propose a novel system, Language-Assisted Continuous Navigation in Structured Spaces (LACNS). LACNS facilitates autonomous driving without the need for HD maps by integrating vehicle-centric local perception with real-time language instructions from map software or human navigators. LACNS begins by generating a BEV map using the vehicle's front-facing camera. Simultaneously, a pretrained Visual Language Model (VLM) detects intersections from the camera images, assigning a score to each. Road elements are then extracted from the BEV map and combined with the intersection scores to identify potential navigation frontiers. Language instructions, processed by a pretrained Large Language Model(LLM), are used to select the most suitable frontier. Finally, the chosen frontier and BEV map are employed to plan a safe route and control the vehicle's movement. We evaluated LACNS using the Carla simulator to validate its navigation capabilities in continuous spaces. Initial experiments involved navigating through four intersections with varying directional instructions, where LACNS demonstrated high and consistent success rates across multiple trials. Further simulations in real-time navigation scenarios revealed that LACNS consistently maintained a high success rate across three progressively challenging routes. These results highlight the effectiveness of our novel autonomous driving navigation method without HD maps.

In recent years, with the rapid development of Advanced Driver Assistance Systems (ADAS), the demand for the precise and efficient surround view stitching system has significantly increased. Traditional stitching methods perform well in small single-unit vehicles with stable camera poses. However, the stitching quality sharply degrades when applied to large tractor-trailers due to the continuous pose changes caused by the non-rigid connection between the tractor and trailer. In detail, first, the extended length of tractor-trailers results in low overlap between cameras, making feature extraction and matching challenging. Additionally, the stitched images often appear irregular, detracting from visual quality. Besides, even if static stitching looks natural, it causes jitter in dynamic scenarios due to random feature extraction. In this paper, we propose an unsupervised deep stitching method for tractor-trailer surround view system. We introduce a feature extraction module for tractor-trailer scenarios (FMT) to enhance feature extraction in low-overlap situations. Besides, we design a spatio-temporally consistent control point constraint strategy (STCC) to achieve spatial shape preservation and temporal smoothing effects, resulting in visually consistent and stable stitched sequences. Experimental results from both public and real dataset show that our method efficiently completes tractor-trailer surround view stitching, producing well-aligned and natural panoramic images compared to previous methods.

Recent advancements in 3D Gaussian Splatting have significantly improved the efficiency and quality of dense semantic SLAM. However, previous methods are generally constrained by limited-category pre-trained classifiers and implicit semantic representation, which hinder their performance in open-set scenarios and restrict 3D object-level scene understanding. To address these issues, we propose OpenGS-SLAM, an innovative framework that utilizes 3D Gaussian representation to perform dense semantic SLAM in open-set environments. Our system integrates explicit semantic labels derived from 2D foundational models into the 3D Gaussian framework, facilitating robust 3D object-level scene understanding. We introduce Gaussian Voting Splatting to enable fast 2D label map rendering and scene updating. Additionally, we propose a Confidence-based 2D Label Consensus method to ensure consistent labeling across multiple views. Furthermore, we employ a Segmentation Counter Pruning strategy to improve the accuracy of semantic scene representation. Extensive experiments on both synthetic and real-world datasets demonstrate the effectiveness of our method in scene understanding, tracking, and mapping, achieving 10× faster semantic rendering and 2× lower storage costs compared to existing methods. Project page: https://young-bit.github.io/opengs-github.github.io/.

Semantic segmentation is a critical technique for effective scene understanding. Traditional RGB-T semantic segmentation models often struggle to generalize across diverse scenarios due to their reliance on pretrained models and predefined categories. Recent advancements in Visual Language Models (VLMs) have facilitated a shift from closedset to open-vocabulary semantic segmentation methods. However, these models face challenges in dealing with intricate scenes, primarily due to the heterogeneity between RGB and thermal modalities. To address this gap, we present Open-RGBT, a novel open-vocabulary RGB-T semantic segmentation model. Specifically, we obtain instance-level detection proposals by incorporating visual prompts to enhance category understanding. Additionally, we employ the CLIP model to assess image-text similarity, which helps correct semantic consistency and mitigates ambiguities in category identification. Empirical evaluations demonstrate that Open-RGBT achieves superior performance in diverse and challenging real-world scenarios, even in the wild, significantly advancing the field of RGB-T semantic segmentation. The project page of Open-RGBT is available at https://OpenRGBT.github.io/.

Automatic Valet Parking (AVP) has garnered significant attention from industry and academia due to its potential to enhance traffic efficiency, parking safety, and user experience. While AVP technologies have been successfully applied in standard parking scenarios with clear markings, real-world parking environments are far more diverse and complex, posing challenges for current systems. To address these limitations, we present Parking-SG, an open-vocabulary hierarchical 3D scene graph representation, facilitating the application of AVP in open and complex environments. Our approach builds an object-based, open-vocabulary map that integrates both ground-level and ground-above objects for comprehensive environmental understanding. Leveraging common sense reasoning and object behavior relationships, various standard or non-standard parking spaces are inferred in open environments. Additionally, we extract and analyze path topology to construct a hierarchical map representation, supporting complex AVP tasks. Parking-SG is validated in both simulated and real-world environments, demonstrating its ability to generate rich environmental representations, accurately and flexibly infer parking spaces, and effectively perform complex AVP tasks.

Image stitching in heavy occlusion scenarios faces the dual challenges of accurate alignment and occlusion removal. On one hand, occlusion causes the loss of key texture and structural information in the image. On the other hand, it affects the image’s integrity. Existing stitching methods perform well in cases with small occlusion coverage, but they often fail in heavy occlusion. This failure is mainly due to three reasons: 1) they cannot identify occluded regions, 2) they cannot suppress interference from the occluded regions, 3) they cannot remove the occluded regions. To address these issues, we propose an unsupervised deep image stitching and de-occlusion method. First, to solve the issue of occluded region identification, we design an Occlusion-Aware Feature Weighted module (OAFW) that explicitly distinguishes between occluded and non-occluded regions by learning the occlusion masks of the images. Second, to address the issue of interference from occlusion, we use the learned occlusion masks to filter out features from the occluded regions. To further suppress the impact of occlusion-induced errors, we design a Mask-Guided Dual-Granularity Alignment loss function (MGDGA) that only calculates alignment errors for non-occluded regions, effectively reducing occlusion error interference during network training. Finally, to resolve the content gap in the occluded regions, we replace the pixels in the occluded areas with those from the aligned overlapping regions and incorporate a Progressive Content Inpainting module (PCI) to recover the missing content in the non-overlapping regions caused by occlusion, ultimately achieving a complete and natural de-occlusion stitched image. Experimental results show that our method improves the mean squared error metric by 17.45% compared to the state-of-the-art stitching method.

In recent years, 3D Gaussian Splatting (3D-GS)based scene representation demonstrates significant potential in real-time rendering and training efficiency. However, most existing methods primarily focus on single-map reconstruction, while the registration and fusion of multiple 3D-GS submaps remain underexplored. Existing methods typically rely on manual intervention to select a reference sub-map as a template and use point cloud matching for registration. Moreover, hard-threshold filtering of 3D-GS primitives often degrades rendering quality after fusion. In this paper, we present a novel approach for automated 3D-GS sub-map alignment and fusion, eliminating the need for manual intervention while enhancing registration accuracy and fusion quality. First, we extract geometric skeletons across multiple scenes and leverage ellipsoid-aware convolution to capture 3D-GS attributes, facilitating robust scene registration. Second, we introduce a multi-factor Gaussian fusion strategy to mitigate the scene element loss caused by rigid thresholding. Experiments on the ScanNet-GSReg and our Coord datasets demonstrate the effectiveness of the proposed method in registration and fusion. For registration, it achieves a 41.9% reduction in RRE on complex scenes, ensuring more precise pose estimation. For fusion, it improves PSNR by 10.11 dB, highlighting superior structural preservation. These results confirm its ability to enhance scene alignment and reconstruction fidelity, ensuring more consistent and accurate 3D scene representation for robotic perception and autonomous navigation.

Recent advancements in 3D Gaussian Splatting(3DGS) have significantly improved semantic scene understanding, enabling natural language queries to localize objects within a scene. However, existing methods primarily focus on embedding compressed CLIP features to 3D Gaussians, suffering from low object segmentation accuracy and lack spatial reasoning capabilities. To address these limitations, we propose GaussianGraph, a novel framework that enhances 3DGS-based scene understanding by integrating adaptive semantic clustering and scene graph generation. We introduce a ‘Control-Follow’ clustering strategy, which dynamically adapts to scene scale and feature distribution, avoiding feature compression and significantly improving segmentation accuracy. Additionally, we enrich scene representation by integrating object attributes and spatial relations extracted from 2D foundation models. To address inaccuracies in spatial relationships, we propose 3D correction modules that filter implausible relations through spatial consistency verification, ensuring reliable scene graph construction. Extensive experiments on three datasets demonstrate that GaussianGraph outperforms state-of-the-art methods in both semantic segmentation and object grounding tasks, providing a robust solution for complex scene understanding and interaction. We provide supplementary video and code at https://wangxihan-bit.github.io/GaussianGraph.

Space-limited U-shape bend is a safety-critical scenario that requires the high maneuverability of vehicles. However, due to the non-holonomic nature of the vehicle, it is difficult to perform flexible U-turns without intricate adjustments, which is detrimental to the efficient execution of tasks. To address these issues, this work incorporates the drifting maneuver of the vehicle and proposes a planning and control framework for time-space efficient passing in constrained U-shape bends. First, a dual-track, 3-Dof vehicle model is developed, incorporating load transfer effects and nonlinear tire forces to enhance trajectory precision. Based on this model, a nonlinear optimization-based planner generates time-optimal, space-efficient, and drift-compatible trajectories while ensuring dynamic feasibility. Finally, a multilayer controller is designed for precise trajectory tracking, integrating a trajectory error feedback compensator, a dynamic state feedforward-feedback regulator, and a model inversion-based actuator controller. Simulation experiments in CarSim validate the proposed framework, demonstrating significant improvements in spatial efficiency and completion time. The results highlight its effectiveness in enhancing autonomous vehicle maneuverability for high-performance applications in constrained environments.

Robotic systems demand accurate and comprehensive 3D environment perception, requiring simultaneous capture of photo-realistic appearance (optical), precise layout shape (geometric), and open-vocabulary scene understanding (semantic). Existing methods typically achieve only partial fulfillment of these requirements while exhibiting optical blurring, geometric irregularities, and semantic ambiguities. To address these challenges, we propose OmniMap. Overall, OmniMap represents the first online mapping framework that simultaneously captures optical, geometric, and semantic scene attributes while maintaining real-time performance and model compactness. At the architectural level, OmniMap employs a tightly coupled 3DGS-Voxel hybrid representation that combines fine-grained modeling with structural stability. At the implementation level, OmniMap identifies key challenges across different modalities and introduces several innovations: adaptive camera modeling for motion blur and exposure compensation, hybrid incremental representation with normal constraints, and probabilistic fusion for robust instance-level understanding. Extensive experiments show OmniMap's superior performance in rendering fidelity, geometric accuracy, and zero-shot semantic segmentation compared to state-of-the-art methods across diverse scenes. The framework's versatility is further evidenced through a variety of downstream applications, including multi-domain scene Q&A, interactive editing, perception-guided manipulation, and map-assisted navigation.

Neural Radiance Fields (NeRF) use multi-view images for 3D scene representation, demonstrating remarkable performance. As one of the primary sources of multi-view images, multi-camera systems encounter challenges such as varying intrinsic parameters and frequent pose changes. Most previous NeRF-based methods assume a unique camera and rarely consider multi-camera scenarios. Besides, some NeRF methods that can optimize intrinsic and extrinsic parameters still remain susceptible to suboptimal solutions when these parameters are poor initialized. In this paper, we propose MC-NeRF, a method for joint optimization of both intrinsic and extrinsic parameters alongside NeRF, allowing individual camera parameters for each image. First, we analyze the coupling issue that arises from the joint optimization between intrinsics and extrinsics, and propose a decoupling constraint utilizing auxiliary images. To further address the degenerate cases in the decoupling process, we introduce an efficient auxiliary image acquisition scheme to mitigate these effects. Furthermore, recognizing that most existing datasets are designed for a unique camera, we provided a new dataset that includes both simulated data and real-world data. Experiments demonstrate the effectiveness of our method in scenarios where each image corresponds to different camera parameters. Specifically, our approach outperforms the baselines favorably in terms of intrinsics estimation, extrinsics estimation, scale estimation, and rendering quality.

Unknown area exploration is a crucial but challenging task for autonomous driving of unmanned ground vehicles (UGV) in unknown off-road environments. However, the exploration efficiency of a single UGV is low due to its limited sensing range. To solve this problem, this paper proposes a risk-inspired aerial active exploration system, which utilizes the flexibility and field of view advantages of Unmanned Aerial Vehicles (UAV) to guide the UGV in unknown off-road environments. Firstly, a fast terrain risk mapping method that can be used for both UAV and UGV is developed. This method efficiently combines quadtree and hash table data structure to enable UAV to analyze large scale terrain point cloud in real time. Based on the risk mapping result, a risk-inspired active exploration method is proposed to actively search a safe reference path for the UGV, which introduces terrain risk information into the process of travel point selection. Finally, the reference path is gradually generated and optimized, so that the UGV can safely and smoothly follow the path to the target location. Compared with single UGV exploration system, our approach reduces the overall path risk by 26.8% in simulated experiments, showing that the proposed system can enhance autonomous driving of the UGV and help it effectively avoid high-risk areas in unknown off-road environments.

In recent years, 3D surround view systems have attracted a lot of attention in the field of advanced driver assistance systems (ADAS). However, the foundational assumption of unchanging camera poses in traditional 3D surround view systems, which is designed for single-unit vehicles, results in a failure to manage the non-rigid connections characteristic of tractor-trailer vehicles. Moreover, tractor-trailer vehicles have the feature of long bodies and large wheelbases, leading to severe distortions and abrupt changes in the rendering results of previous 3D texture mapping models. In this paper, we propose DSVT, a dynamic 3D surround view system for tractor-trailer vehicles, designed to address the aforementioned issues. Specifically, we develop a dynamic surround image stitching algorithm based on relative pose estimation, which estimates the relative poses between cameras and stitches all images together to generate a 2D panoramic image. Subsequently, a novel 3D drop model is proposed, mapping the 2D panoramic image onto the 3D model for panoramic viewing. Our system can run in real time on Nvidia AGX Orin. Experimental results in real tractor-trailer scenes show that our system can achieve more accurate and natural visual effects.

Self-supervised depth estimation has attracted much attention due to its ability to improve the 3D perception capabilities of unmanned systems. However, existing unsupervised frameworks rely on the assumption of photometric consistency, which may not hold in challenging environments such as night-time, rainy nights, or snowy winters due to complex lighting and reflections, resulting in inconsistent photometry across different frames for the same pixel. To address this problem, we propose a self-supervised monocular depth estimation unified framework that can handle these complex scenarios, which has the following characteristics: (1) an Illumination Compensation PoseNet (ICP) is designed, which is based on the classic Phong illumination theory and compensates for lighting changes in adjacent frames by estimating per-pixel transformations; (2) a Dual-Axis Transformer (DAT) block is proposed as the backbone network of the depth encoder, which infers the depth of local repeat-texture areas through spatial-channel dual-dimensional global context information of images. Experimental results demonstrate that our approach achieves state-of-the-art depth estimation results in complex environments on the challenging Oxford RobotCar dataset.

Recently, Bird’s Eye View (BEV) detection methodologies that utilize surround-view cameras have seen significant advancements in autonomous driving systems. Traditional methods, however, are constrained by their reliance on specific camera parameters, which poses challenges in generalizing across different vehicle-mounted cameras with varying poses and under adverse conditions. To address these challenges, we propose a robust BEV representation network that integrates Dual-Space Positional Encoding (DSPE) and image perception. This network is designed to enhance resilience to calibration errors and pose fluctuations, resulting in reliable detection performance on the Nuscenes dataset, even with imprecise extrinsic inputs. Our approach demonstrates competitive accuracy when compared to other methods that do not rely on temporal data, highlighting the effectiveness of our DSPE strategy in improving the robustness and accuracy of BEV detection in dynamic and challenging environments.

3D reconstruction methods such as Neural Radiance Fields (NeRFs) are capable of optimizing high-quality 3D representation from images. However, NeRF is limited by the requirement for a large number of multi-view images, making its application to real-world scenarios challenging. In this work, we propose a method that can reconstruct real-world scenes from a few input images and a simple text prompt. Specifically, we fine-tune a pretrained diffusion model to constrain its powerful priors to the visual inputs and generate 3D-aware images, leveraging the coarse renderings obtained from input images as the image condition, along with the text prompt as the text condition. Our fine-tuning method saves a significant amount of training time and GPU memory usage while also generating credible results. Moreover, to enable our method to have self-evaluation capabilities, we design a semantic switch to filter out generated images that do not match real scenes, ensuring that only informative priors from the fine-tuned diffusion model are distilled into the 3D model. The semantic switch we designed can be used as a plug-in and improve performance by 13%. We perform our approach on a real-world dataset and demonstrate competitive results compared to existing sparse-view 3D reconstruction methods. Please see our project page for more visualizations and code: https://bityia.github.io/FDfusion.

Real time and highly robust localization is essential for location-based services and autonomous driving. Nevertheless, it is hard to obtain high-quality observations from these vehicle-level positioning sensors because of the uncertainty of urban environment and conditions, which affects the localization performance. In this study, we propose an adaptive optimal selection-robust hybrid adaptive Kalman filter (AOS-RHAKF) method of combination data from BeiDou navigation satellite system (BDS)/the fifth-generation (5G) network to achieve high-accuracy positioning estimation in urban complex environment. The proposed method is mainly composed of three sequential modules, namely, initial positioning estimation, AOS-based 5G base stations (BSs) measurement data optimization and BDS/5G combined positioning. Initial positioning estimation uses the raw measurement data and the basic mathematical model with position estimation to work out the mobile vehicle position. The AOS-based 5G BSs measurement data optimization module achieves better reselection of observation data through the adaptive optimal selection factor. The BDS/5G combined positioning method utilizes the optimized 5G data and BDS to establish a tightly coupled structure model, and then achieves high-precision positioning of mobile vehicles using RHAKF method. Finally, both simulations and actual driving test were carried out. The results show that the proposed AOS-RHAKF method significantly improves the positioning accuracy compared with the BDS, 5G-only, and BDS/5G loose coupling positioning using the raw measurement data.

All-day self-supervised monocular depth estimation has strong practical significance for autonomous systems to continuously perceive the 3D information of the world. However, night-time scenes pose challenges of weak texture and violating the brightness consistency assumption due to low illumination and varying lighting, respectively, which easily leads to most existing self-supervised models only being able to handle day-time scenes. To address this problem, we propose a self-supervised monocular depth estimation unified framework that can handle all-day scenarios, which has three features: (1) an Illumination Compensation PoseNet (ICP) is designed, which is based on the classic Phong illumination theory and compensates for lighting changes in adjacent frames by estimating per-pixel transformations; (2) a Dual-Axis Transformer (DAT) block is proposed as the backbone network of the depth encoder, which infers the depth of local low-illumination areas through spatial-channel dual-dimensional global context information of night-time images; (3) a cross-layer Adaptive Fusion Module (AFM) is introduced between multiple DAT blocks, which learns attention weights between different layer features and adaptively fuses cross-layer features using the learned weights, enhancing the complementarity of different layer features. This work was evaluated on multiple datasets, including: RobotCar, Waymo and KITTI datasets, achieving state-of-the-art results in both day-time and night-time scenarios.

Drivers’ cognitive mechanisms could benefit the development of human-like automated driving (AD) strategies, which are with high intelligence and comfort levels. The common approach of human-like AD is to learn from human demonstration data, for which it is exhausting and difficult to construct well-rounded and reliable datasets. Therefore, we proposed the human-like AD decision-making method based on drivers’ cognition mechanism. The fundamental and difficult thing of this method is to figure out drivers’ cognitive mechanism systematically and comprehensively, which is still either too rough or too fragmented for AD development. By integrating the abundant studies about drivers’ cognition in multiple fields, we propose two novel conceptual models: Potential Hazard Model (PHM) illustrates the mechanisms of drivers’ reaction in the simple meta-scenarios while Candidate Selection Model (CSM) explains how drivers handle complicated scenarios based on PHM. Based on PHM and CSM, we propose the human-like decision-making method for AD. This method integrates cognitive mechanisms, natural driving data, optimization-based planning, and other techniques profoundly. The experiments in extensive road and traffic scenarios verify that the method show good generalizability, interpretability, and human-likeness.

Prediction is a vital component of motion planning for autonomous vehicles (AVs). By reasoning about the possible behavior of other target agents, the ego vehicle (EV) can navigate safely, efficiently, and politely. However, most of the existing work overlooks the interdependencies of the prediction and planning module, only connecting them in a sequential pipeline or underexploring the prediction results in the planning module. In this work, we propose a framework that integrates the prediction and planning module with three highlights. First, we propose an ego-conditioned model for causal prediction, with the introduced edge-featured graph transformer model, the impact the ego future maneuver poses to the target vehicles is demonstrated. Second, we develop a motion planner based on ‘dynamic voxels’ in the spatio-temporal domain, enabling the time-to-collision criterion evaluation and the optimal trajectory generation in continuous space. Third, the prediction and planning modules are coupled in a closed-loop and efficient form. Specifically, taking each maneuver as a cluster, representative trajectory primitives are generated for conditional prediction, and conversely, prediction results are used to score the primitives as guidance, which alleviates the duplicated callback of the prediction module. The simulations are conducted in overtaking, merging, unprotected left turns, and also scenarios with imperfect social behaviors. The comparison studies demonstrate the better safety assurance and efficiency of the proposed model, and the ablation experiments further reveal the effectiveness of the new ideas.

Automotive surround-view camera systems have been commonly employed in automated driving to aid in near-field sensing and other perception tasks. Due to the large size of the body and the presence of multiple blind spots, panoramic surround-view systems are particularly crucial for tractor-trailer vehicles. However, the non-rigid body of tractor-trailer vehicles introduces pose changes between cameras, rendering traditional calibration-based methods inadequate. Additionally, cameras mounted separately on the tractor and the trailer will experience independent vibrations, resulting in undesirable shakiness in captured videos. In this paper, we propose a unified video stabilization and stitching method to address these challenges, which can smooth the unsteady frames and align the images from moving cameras. Delving into video stabilization techniques, we extend mesh-based motion model for unified stitching and leverage deep-learning based modules to handle complex real-world scenarios. Moreover, we design a new optimization framework to estimate the optimal displacements of mesh vertices, enabling simultaneous stabilization and stitching of frames. The experimental results, obtained by public datasets and videos captured from a model tractor-trailer vehicle, demonstrate that our approach outperforms previous methods and is highly effective in real-world applications.

Self-supervised monocular depth estimation methods have achieved remarkable results on natural clear images. However, it is still a serious challenge to directly recover depth information from blurred images caused by long-time exposure while camera fast moving. To address this issue, we propose a unified framework for simultaneous deblurring and depth estimation (SDDE), which has higher coupling performance and flexibility compared with the simple concatenation strategy of deblurring model and depth estimation model. This framework mainly benefits from three features: 1) a novel Task-aware Fusion Module (TFM) to adaptively select the most relevant intermediate shared features for the dual decoder network by aggregating multi-scale features, 2) a unique Spatial Interaction Module (SIM) to learn higher-order representation in the encoder stage to better describe complex boundaries of different classes in high-dimensional space, and focuses on the task-related region by modeling the pairwise spatial correlation of the holistic tensor, 3) a Priors-Based Composite Regularization term to jointly optimize the shared encoder-dual decoder network. This work was evaluated on multiple datasets, including: Stereo blur, KITTI,NYUv2, REDS and our own large-scale stereo blur dataset, resulting in state-of-the-art results for depth estimation and image deblurring, respectively.

Uncontrolled intersections with interaction and uncertainties are challenging for autonomous vehicles (AV) to manage. In this work, we propose a decision-making model specific to intersections with emphasis on three aspects. First, behavior estimation of the social vehicles’ (SVs) is essential for risk avoidance. We try to improve prediction accuracy by predicting the intentions and driving styles of SVs in advance and doing adaptive goal sampling. Second, the uncertainty from the prediction results should be considered in the decision-making process. For this, a risk-aware framework is developed, composed of a Subordinate Driver (SD) and a Primary Driver (PD) for decision-making and planning. Particularly, in SD, the prediction-guided strategy tree is built to search for an optimal strategy with observation and action branch trimming, which employs the prediction results for risk assessment. In PD, to mimic the both-way negotiation among vehicles, the level-k game model is deployed to determine the action in the players’ best interest and update the estimation of driving styles. Third, the generated maneuver is required to be evaluated in a closed-loop simulation. A ‘semi-autonomous’ control model is designed, which is a combination of the dataset and the stochastic sampling model. The results of ablation experiments verify the function of each module. The case studies and comparison experiments demonstrate the effectiveness of the framework in highly interactive intersections.

Fisheye object detection is a difficult task in robotics and autonomous driving. One of the reasons is that the fisheye datasets are inferior to standard image datasets in scale and quantity, which inspires the idea of using standard image datasets for fisheye object detection. However, the models trained on standard image datasets do not perform well with fisheye data. In this work, we explore the effect of fisheye images on different stages of the YOLOX with published weights generated by standard image datasets. We also propose a new regression strategy for 24-points object representation method, which is insensitive to image distortion. The experiments show that the feature extraction part is robust to fisheye image features, while the regression part of location and category performs poorly. The strategy can achieve the position of discrete points without calculating the IOU of irregular-shaped boxes. Theoretically, the strategy can be widely adopted to regress the irregular bounding boxes composed of discrete points. Source code is at https://github.com/IN2-ViAUn/Exploration-of-Potential.

The in-depth application of the IoT technology in the gas industry has improved the intelligence of the gas system. As an important part of the gas system, the optimization of state monitoring of regulator stations is of great significance. The efficiency of monitoring can be improved by feature selection, but the reduction of important features will reduce the accuracy of identification. Therefore, a feature selection method based on gray relational analysis is proposed. First, the distance between the comparison matrix and the reference matrix is transformed using the geometric probability distribution, which solves the problem of data initial transformation. Then, an adaptive value method of resolution coefficient based on the cuckoo algorithm is proposed. Finally, a weight coefficient is constructed by combining the redundancy and the correlation to improve the gray relational degree applied to nonsequential systems. The maximum classification accuracy and the number of selected features are used to compare the performance of feature selection methods for several different types of data sets. Simulation analysis shows that the proposed method has higher maximum classification accuracy and smaller selected feature set. Finally, the proposed method is applied to the state monitoring of gas regulator stations. Seven feature selection methods, including the proposed method, the classical methods, and the advanced methods are combined with a support vector machine,$k$-nearest neighbor, and decision tree, respectively. The experimental results indicate that the comprehensive performance of the proposed method combined with the three classifiers is good. It enables the subset containing the most identifiable features to be obtained quickly, which is beneficial to improve the efficiency of state monitoring.

Autonomous driving, including intelligent decision-making and path planning, in dynamic environments (like highway) is significantly more difficult than the navigation in static scenarios because of the additional time dimension. Therefore, correlating the time dimension and the space dimension through prediction to create a spatio-temporal navigation map can make decision-making and path planning in such kinds of environment much easier. In this article, NGSIM data is analysed and processed from the perspective of the ego-vehicle (using the data as an ego-vehicle’s perception results). Based on the data, we develop an LSTM (Long-Short Term Memory)-based framework to predict possible trajectories of multiple surrounding vehicles within a certain range of the ego-vehicle. Then, the multiple predicted trajectories in a series of continuous dynamic highway scenes are projected into a spatio-temporal domain to create an octree map. Thus, dynamic targets and static obstacles can be unified into the same domain or map so that the dynamic disturbance problem for autonomous driving in highway environments can be resolved. Experimental results show that the proposed model is capable of predicting all the future trajectories around the ego-vehicle efficiently and the corresponding spatio-temporal map can be generated accurately in different dynamic scenarios.

As the vital factor of vehicle behavior understanding and prediction, vehicle taillight recognition is an important technology for autonomous driving, especially in diverse actual traffic scenes full of dynamic interactive traffic participants. However, in practical application, it always faces many challenges, such as ‘variable lighting conditions’, ‘non-uniform taillight standards’ and ‘random relative observation pose’, which lead to few mature solutions in current common autopilot systems. This work proposes an action-state joint learning-based vehicle taillight recognition method on the basis of vehicles detection and tracking, which takes both taillight state features and time series features into account, consequently getting practicable results even in complex actual scenes. In detail, vehicle tracking sequence is used as input and split into pieces through a sliding window. Then, a CNN-LSTM model is applied to simultaneously identify the action features of brake lights and turn signals, dividing taillight actions into five categories: None, Brake_on, Brake_off, Left_turn, Right_turn. Next, the brightness of high-position brake light is extracted through semantic segmentation and combined with taillight actions to form higher-level features for taillight state sequence analysis. Finally, an undirected graph model is used to establish the long-term dependence between successive pieces by analysing the higher-level features, thus inferring the continuous taillight state into:$off$,$brake$,$left$,$right$. Datasets including daytime, nighttime, congested road, highway, etc. were collected, tested and published in our work to demonstrate its effectiveness and practicability.

Trajectory planning for the unmanned vehicle in the complex environment has always been a challenging task. Planned trajectory with the corresponding target velocity or acceleration sequence must be collision-free guaranteed and as comfortable as possible on the premise of obeying the traffic rules and interaction with other dynamic social vehicles. To meet this requirement, this paper proposes a framework for trajectory planning based on spatio-temporal map. Due to the time layer architecture in the map, the trajectory can be generated with velocity and acceleration simultaneously, and the whole trajectory is constrained within a ‘safety strip’, resulting in an efficient and safety guaranteed trajectory. The framework is composed of three sections: rough search, fine optimization and safety strip-based collision avoidance. For rough search, we propose an improved A* algorithm implemented in the discrete time layer to find out the suboptimal states efficiently. In fine optimization, the B-spline curve is exploited to connect the searched states into a continuous trajectory. And the optimal control points of B-spline are further grouped into several segments, forming the safety strip which is actually the distribution space of the planned trajectory. If necessary, an adjustment will be applied to keep the strip away from the collision zone, making the entire trajectory completely collision-free. Experiments on both public dataset and self-driving simulator show that the proposed framework can adapt to different kinds of complex traffic scenes well.

Intelligent decision making and efficient trajectory planning are closely related in autonomous driving technology, especially in highway environment full of dynamic interactive traffic participants. This work integrates them into a unified hierarchical framework with long-term behavior planning (LTBP) and short-term dynamic planning (STDP) running in two parallel threads with different horizon, consequently forming a closed-loop maneuver and trajectory planning system that can react to the dynamic environment effectively and efficiently. In LTBP, a novel voxel structure and the ‘voxel expansion’ algorithm are proposed for the generation of driving corridors in 3D configuration, which involves the prediction states of surrounding vehicles. By using Dijkstra search, the maneuver with minimal cost is determined in form of voxel sequences, then a quadratic programming (QP) problem is constructed for solving the optimal trajectory. And in STDP, another small-scaled QP problem is performed to track or adjust the reference trajectory from LTBP in response to the dynamic obstacles. Meanwhile, a Responsibility-Sensitive Safety (RSS) Checker keeps running at high frequency for real-time feedback to ensure security. Experiments on real data collected in different highway scenarios demonstrate the effectiveness and efficiency of our work.

Trajectory prediction for vehicles is a popular subject since it is beneficial for efficient and secure trajectory planning. In structured traffic scenarios, the behaviour and motion of vehicles are heavily dependent on the social interaction constraints, such as road geometry and surrounding vehicles, and the kinematics model constraints, such as continuous heading and maximum acceleration. To take these factors into account, we analyse the particular characteristics of driving vehicles and propose a model that predicts the possible and feasible trajectory for host vehicle in 3 seconds. In this model, the trajectory of host vehicle takes the center-line as reference, imitates the leader vehicle and focuses on the social vehicles through attention concentration mechanism (ACM) with spatial and temporal information encoded in a fusion hidden state. Furthermore, in order to make the trajectory feasible for vehicle dynamics and kinematics, we introduce a prediction diagnosis method to check the continuous heading and maximum acceleration condition, pruning and adjusting the prediction candidates. Experiments on released public datasets show that this framework can well evaluate the traffic interactions and forecast the trajectory more accurately than common networks.

An optimal distributed fusion estimation problem is concerned in this study for a kind of linear dynamic multirate sensors systems with correlated noise and stochastic unreliable measurements. The system is formulated at the finest scale with multiple sensors at different scales observing a common target independently with different sampling rates. The noise between different sensors is relevant, moreover, is also correlated with the system noise. The authors derive the local state estimation algorithms under the circumstance of total reliable measurements and stochastic unreliable measurements occur occasions, and the optimal distributed Kalman filter fusion algorithm, respectively. The authors provide a simulation example to illustrate the effectiveness and feasibility of the proposed algorithm.

This paper presents real-time obstacles detection and their status classification method for collision warning in the vehicle active safety system. Specifically, stereo cameras and millimeter wave (mmw)-radar are fused to help the driving ego-vehicle to find “Danger” or “Potential Danger” in a timely way through combining with the vehicle kinematic model. The proposed method makes full use of the unique advantages of stereo cameras and mmw-radar to sense the environment through several modules. Cameras are mainly used to detect the near or lateral dynamic objects and to obtain the obstacles region of interest (ROI) considering its rich information and high sensitivity to the lateral displacement, while far or longitudinal relative dynamic objects are detected by mmw-radar according to its observational ability to make up for the disadvantage of cameras. In detail, a cameras detector utilizes ”error vectors” rather than the optical flow to obtain dynamic classes through two times clustering. Mmw-radar mainly detects relative dynamic objects, whose absolute speed can be computed according to the ego-vehicle's state. Then, the detected objects of these two detectors are integrated in an obstacles ROI map, which is obtained through an UV-disparity obstacles detection algorithm to get the final dynamic and relative dynamic objects. Finally, they are classified by comparing them with a dangerous area that is acquired according to the vehicle kinematic model in a special vehicle coordinate system, which is fixed to the ground temporarily. This method is tested on our mobile platforms and the results prove that it can work effectively even though the ego-vehicle drives quickly.

As communication bandwidth and resources are limited in network-based control systems, in order to reduce superfluous waste, it is necessary to design an event-triggered communication mechanism. In this paper, the problem of event-triggered state estimation is studied for fusion of multiple sensors with correlated noise. The noise of different sensors are cross-correlated and coupled with the system noise of the previous step and the same time step. An optimal state estimation algorithm based on iterative estimation of white noise estimator is presented, which makes full use of the observation information effectively. A numerical example is used to illustrate the effectiveness of the presented algorithm.

This paper presents a real-time and robust lane detection and forward collision warning technique based on stereo cameras. First, obstacles image is obtained through stereo matching and UV-disparity segmentation algorithm. Then, Inverse Perspective Mapping(IPM) and Sobel filtering are conducted to generate a low-noise top view of the road by fusing the obstacles image and the original image. Next, Hough Transformation for the top view map is completed and the extreme points(poles) are calculated as the detected lanes according to the traffic lanes model. Besides, the host lane is selected or supplemented among all the detected lanes and the nearest obstacle in this host lane is detected for the forward collision warning. Experimental results on the public data set indicate that our method can work effectively and real-timely in the normal structured environment.

This paper describes an improved decision and planning algorithm based on our previously proposed methods for unmanned ground vehicle (UGV). The new method can be applied to UGV driving both in structured environment and unstructured environment. In the improved method, the prospect of planning is extended from 40m to 100m for safe driving at high speed and some piecewise linear speed functions are designed for the new prospect. After this improvement our UGV now can drive at a maximum speed of 60km/h rather than 40km/h while avoiding obstacles safely. Besides, a velocity feedforward control is added to make the UGV overtake other cars driving at about 25km/h on the road. At last, the collision detection algorithm is improved to make the lane changing maneuver safer. The proposed decision and planning algorithm is implemented both on our old Polaris all terrain vehicle (ATV) and new FAW-H7 car, which exhibited good performance on Across Dangers & Obstacles 2016, Tahe, China and Future Challenge 2016, Changshu, China, respectively.

Obstacle detection plays an important role for robot collision avoidance and motion planning. This paper focuses on the study of the collision prediction of a dual-arm robot based on a 3D point cloud. Firstly, a self-identification method is presented based on the over-segmentation approach and the forward kinematic model of the robot. Secondly, a simplified 3D model of the robot is generated using the segmented point cloud. Finally, a collision prediction algorithm is proposed to estimate the collision parameters in real-time. Experimental studies using the Kinect Ⓡ sensor and the Baxter Ⓡ robot have been performed to demonstrate the performance of the proposed algorithms.

This paper proposes a stable adaptive fuzzy fault-tolerant attitude-tracking controller for a rigid spacecraft in the presence of unavailable velocities, external disturbance, actuator faults, and actuator saturation. Fuzzy logic systems are applied to approximate the unknown nonlinear function vector, and a fuzzy adaptive observer is designed to estimate the unmeasured velocity of the rigid body. By using the backstepping technique, a novel adaptive fuzzy attitude-tracking fault-tolerant control scheme is developed. It has been testified that this control approach guarantees that all signals of the rigid spacecraft are bounded and the tracking error between the system output and the reference signal converges to a small neighborhood of zero. Simulation examples with constant faults and time-variant faults are provided to show the fault-tolerant effectiveness of the control method.

This paper describes a novel amphibious robot, which adopts a dual-swing-legs propulsion mechanism, proposing a new locomotion mode. The robot is called FroBot, since its structure and locomotion are similar to frogs. Our inspiration comes from the frog scooter and breaststroke. Based on its swing leg mechanism, an unusual universal wheel structure is used to generate propulsion on land, while a pair of flexible caudal fins functions like the foot flippers of a frog to generate similar propulsion underwater. On the basis of the prototype design and the dynamic model of the robot, some locomotion control simulations and experiments were conducted for the purpose of adjusting the parameters that affect the propulsion of the robot. Finally, a series of underwater experiments were performed to verify the design feasibility of FroBot and the rationality of the control algorithm.

For the local path planning problem of autonomous vehicle in a complicated environment, a method combining cubic hermite spline curves with the kinematic model of autonomous vehicle is developed. And a novel algorithm for obstacle avoidance, called navigation circle, is proposed to take the road structure into account, which is a practical method for real-time path planning. In the new method, one of the trajectory generated by cubic hermite spline curves or navigation circle is optimized through the kinematic model of autonomous vehicle to get the kinematically feasible trajectory. The optimization is actually a numerical forward propagation and is easy to implement. The simulation experiment is conducted on the Robot Operating System (ROS) platform, which is based on replaying the data of the real world obtained from sensors or other modules on autonomous vehicle. Satisfactory simulation results verify the validity and the efficiency of the proposed method as well as the planner's capability to navigate in a realistic scenario.

This paper describes our preliminary development of a virtual robot teleoperation platform based on hand gesture recognition using visual information. Hand gestures in images captured by a camera are recognised to control a virtual iCub. We employ two methods to realise the classification: Adaptive Neuro-fuzzy Inference Systems (ANFIS) and Support Vector Machines (SVM). We realise the teleoperation of a virtual robot using iCubSimulator. The technique in the paper will enable us to teleoperate a physical robot in the future work. In addition, a video server is set up to monitor the real robot. By using the parallel system we are able to improve the robot's performance. Based on the techniques presented in this paper, the virtual iCub can perform the specified actions remotely in a natural manner.

In this paper, in order to combine intelligence of human operator and automatic function of the robot, we design a control scheme for the bimanual robot manipulation, in which the leading robot arm is directly manipulated by a human operator through a haptic device and the following robot arm will automatically adjust its motion to match the operator's motion. In this paper, we propose a fuzzy-based adaptive feedforward compensation controller and apply it into the robot control. According to the comparison results in the simulated experiment, we conclude that the fuzzy-adaptive controller performs better than the non-fuzzy controller, although they can both complete the specified task by tracking the leading robot arm controlled by the human operator. The techniques developed in this paper could be very useful for our future study on adaptation in human-robot interaction in improving the reliability, safety and intelligence.

We proposed an unsupervised algorithm to extract profile features and detect moving object under dynamic background in 3D range Data. Moving object detection under dynamic background has become an increasingly popular research topic in mobile robotics. For the characteristics of dynamic background scene, we proposed an online unsupervised moving object detection algorithm, based on Gaussian Mixture Models and Motion Compensation. Furthermore, we did the work of clustering and identifying of the targets. In order to improve the robustness of the algorithm, we used a tracker to track the results of the detection. At last, experimental results on real laser data depicting urban and rural scenes under static and dynamic background are presented.

Formation control problem has been extensively investigated in the literature of multi-agent systems, robotics, and control, etc. Our previous work mainly concentrated on the theoretic study of line formation with three robots, however, it is hard to handle with the formation problem whose number of robots is strictly larger than 3. In order to effectively overcome this problem, this paper incorporates convex hull and the standard PSO algorithm to design the typical formation of several robots. Firstly, on the basis of convex hull of robots, objective function, corresponding to several constraints, is given by the new convex hull method. Secondly, the standard PSO algorithm is adopted to search for the desired positions of several robots to minimize the objective function and satisfy the formation constraints. To demonstrate the effectiveness of the proposed algorithm, numerical results, regarding the formation of several ships in the realistic ocean, mainly concentrate on triangle formation, diamond formation and regular polygon formation.

In this paper, a kind of integrated perception system for mobile robot is presented, which consists of 3D Lidar, 2D camera and their spatial registration. Based on the system and support vector machine (SVM), a self-supervised learning scheme between 3D point cloud data and 2D image data has been established, which can identify the traversable lane in driving environments through data association and parameters training. With this approach, vision-based autonomous navigation can be achieved and its effectiveness has been verified by extensive robot experiments.

This paper considers a quantized system with finite-level quantized input computed from quantized measurements (QIQM). The problem of globally asymptotic stability of QIQM system is transferred to the one of an equivalent system depending on a multiplier which is nonnegative and bounded. It is the focus of this paper to discuss the nonnegativity and boundness of the multiplier. A sufficient condition is given for the globally asymptotic stability of QIQM system. Note the main method used here is similar to the one used by Richter and Misawa, in which the uniform quantizer is employed. This paper is the extension of that work to the logarithmic quantization which is more advantage than the uniform one. A numerical simulation is presented at last to show the effectiveness of the main results and the advantage of logarithmic quantization compared to uniform quantization.

This paper presents the design, analysis and performance of modified continuous extended Kalman filter based on complementary properties. The modified design departs from traditional solutions as better dynamic performance and steady-state performance for estimating the attitude angles on three axises, and effectively dominate the disturbance in practical system. In order to illustrate the better performance, the simulation results are given by comparing traditional extended Kalman filter with modified filter.

Wireless sensor network localization is an essential problem that has attracted increasing attention due to wide requirements such as in-door navigation, autonomous vehicle, intrusion detection, and so on. With the a priori knowledge of the positions of sensor nodes and their measurements to targets in the wireless sensor networks (WSNs), i.e. posterior knowledge, such as distance and angle measurements, it is possible to estimate the position of targets through different algorithms. In this contribution, two approaches based on least-squares and Kalman filter are described for localization of one static target in the WSNs with distance, angle, or both distance and angle measurements, respectively. Noting that the measurements of these sensors are generally noisy of certain degree, it is crucial and interesting to analyze how the accuracy of localization is affected by the sensor errors and the sensor network, which may help to provide guideline on choosing the specification of sensors and designing the sensor network. To this end, we make theoretical analysis for the different methods based on three types of measurement noise: bounded noise, uniformly distributed noises, and Gaussian white noises. Simulation results illustrate the performance comparison of these different methods.

With the development of wireless communication technology, wireless network has been introduced into networked control system named WNCS. The wireless network brings some challenging issues. Here we consider three sources: ambient wireless traffic, block fading, and multipath fading. Based on the mathematical models of them, a wireless network is emulated. Networked predictive control (NPC) is then presented and used to make an inverted pendulum stable in the simulation. In the Simulink diagram the wireless network modules, the predictive controller, and the compensator are all implemented by C-MEX s-function. And the simulation result shows the effectiveness of the control scheme.

In “Comments on Constrained Infinite-Horizon Model Predictive Control for Fuzzy-Discrete-Time Systems” (IEEE Trans. Fuzzy Syst.), Ding criticized that the application of Lyapunov method, the handling of the output constraint, and the resultant linear-matrix-inequality-optimization problem are incorrect in IEEE Trans. Fuzzy Syst. , vol. 18, no. 2, pp. 429-436, Apr. 2010. This rebuttal answers some key points raised in the comments.

In this paper, a 3D laser point cloud-based navigation method for autonomous ground vehicles in complex environment is proposed. With a coordinate transformation of laser data from sphere to cylinder, environment perception cylinder is configured. In the cylinder, terrain traversability is predicted through analysis on radial and tangential slope of 3D point cloud. In addition, the candidate point cloud of traversable region has been extracted. Furthermore, navigation circle contained with direction information is built up based on the candidate point cloud. Extended experimental results demonstrate that the method allows autonomous ground vehicle to move safely and correctly in complex environment.

The problem of constrained infinite-horizon model-predictive control for fuzzy-discrete systems is considered in this paper. New sufficient conditions are proposed in terms of linear-matrix inequalities. Based on the optimal solutions of these sufficient conditions at each sampling instant, we design both parallel-distributed compensation and nonparallel-distributed compensation state-feedback controllers, which can guarantee that the resulting closed-loop fuzzy-discrete system is asymptotically stable. In addition, the fuzzy-feedback controllers meet the specifications for the fuzzy-discrete systems with both input and output constraints. Numerical examples are presented to demonstrate the effectiveness of the proposed techniques.