In recent years, with the rapid development of Advanced Driver Assistance Systems (ADAS), the demand for the precise and efficient surround view stitching system has significantly increased. Traditional stitching methods perform well in small single-unit vehicles with stable camera poses. However, the stitching quality sharply degrades when applied to large tractor-trailers due to the continuous pose changes caused by the non-rigid connection between the tractor and trailer. In detail, first, the extended length of tractor-trailers results in low overlap between cameras, making feature extraction and matching challenging. Additionally, the stitched images often appear irregular, detracting from visual quality. Besides, even if static stitching looks natural, it causes jitter in dynamic scenarios due to random feature extraction. In this paper, we propose an unsupervised deep stitching method for tractor-trailer surround view system. We introduce a feature extraction module for tractor-trailer scenarios (FMT) to enhance feature extraction in low-overlap situations. Besides, we design a spatio-temporally consistent control point constraint strategy (STCC) to achieve spatial shape preservation and temporal smoothing effects, resulting in visually consistent and stable stitched sequences. Experimental results from both public and real dataset show that our method efficiently completes tractor-trailer surround view stitching, producing well-aligned and natural panoramic images compared to previous methods.

Neural Radiance Fields (NeRF) use multi-view images for 3D scene representation, demonstrating remarkable performance. As one of the primary sources of multi-view images, multi-camera systems encounter challenges such as varying intrinsic parameters and frequent pose changes. Most previous NeRF-based methods assume a unique camera and rarely consider multi-camera scenarios. Besides, some NeRF methods that can optimize intrinsic and extrinsic parameters still remain susceptible to suboptimal solutions when these parameters are poor initialized. In this paper, we propose MC-NeRF, a method for joint optimization of both intrinsic and extrinsic parameters alongside NeRF, allowing individual camera parameters for each image. First, we analyze the coupling issue that arises from the joint optimization between intrinsics and extrinsics, and propose a decoupling constraint utilizing auxiliary images. To further address the degenerate cases in the decoupling process, we introduce an efficient auxiliary image acquisition scheme to mitigate these effects. Furthermore, recognizing that most existing datasets are designed for a unique camera, we provided a new dataset that includes both simulated data and real-world data. Experiments demonstrate the effectiveness of our method in scenarios where each image corresponds to different camera parameters. Specifically, our approach outperforms the baselines favorably in terms of intrinsics estimation, extrinsics estimation, scale estimation, and rendering quality.

In recent years, 3D surround view systems have attracted a lot of attention in the field of advanced driver assistance systems (ADAS). However, the foundational assumption of unchanging camera poses in traditional 3D surround view systems, which is designed for single-unit vehicles, results in a failure to manage the non-rigid connections characteristic of tractor-trailer vehicles. Moreover, tractor-trailer vehicles have the feature of long bodies and large wheelbases, leading to severe distortions and abrupt changes in the rendering results of previous 3D texture mapping models. In this paper, we propose DSVT, a dynamic 3D surround view system for tractor-trailer vehicles, designed to address the aforementioned issues. Specifically, we develop a dynamic surround image stitching algorithm based on relative pose estimation, which estimates the relative poses between cameras and stitches all images together to generate a 2D panoramic image. Subsequently, a novel 3D drop model is proposed, mapping the 2D panoramic image onto the 3D model for panoramic viewing. Our system can run in real time on Nvidia AGX Orin. Experimental results in real tractor-trailer scenes show that our system can achieve more accurate and natural visual effects.

Recently, Bird’s Eye View (BEV) detection methodologies that utilize surround-view cameras have seen significant advancements in autonomous driving systems. Traditional methods, however, are constrained by their reliance on specific camera parameters, which poses challenges in generalizing across different vehicle-mounted cameras with varying poses and under adverse conditions. To address these challenges, we propose a robust BEV representation network that integrates Dual-Space Positional Encoding (DSPE) and image perception. This network is designed to enhance resilience to calibration errors and pose fluctuations, resulting in reliable detection performance on the Nuscenes dataset, even with imprecise extrinsic inputs. Our approach demonstrates competitive accuracy when compared to other methods that do not rely on temporal data, highlighting the effectiveness of our DSPE strategy in improving the robustness and accuracy of BEV detection in dynamic and challenging environments.

Automotive surround-view camera systems have been commonly employed in automated driving to aid in near-field sensing and other perception tasks. Due to the large size of the body and the presence of multiple blind spots, panoramic surround-view systems are particularly crucial for tractor-trailer vehicles. However, the non-rigid body of tractor-trailer vehicles introduces pose changes between cameras, rendering traditional calibration-based methods inadequate. Additionally, cameras mounted separately on the tractor and the trailer will experience independent vibrations, resulting in undesirable shakiness in captured videos. In this paper, we propose a unified video stabilization and stitching method to address these challenges, which can smooth the unsteady frames and align the images from moving cameras. Delving into video stabilization techniques, we extend mesh-based motion model for unified stitching and leverage deep-learning based modules to handle complex real-world scenarios. Moreover, we design a new optimization framework to estimate the optimal displacements of mesh vertices, enabling simultaneous stabilization and stitching of frames. The experimental results, obtained by public datasets and videos captured from a model tractor-trailer vehicle, demonstrate that our approach outperforms previous methods and is highly effective in real-world applications.

Fisheye object detection is a difficult task in robotics and autonomous driving. One of the reasons is that the fisheye datasets are inferior to standard image datasets in scale and quantity, which inspires the idea of using standard image datasets for fisheye object detection. However, the models trained on standard image datasets do not perform well with fisheye data. In this work, we explore the effect of fisheye images on different stages of the YOLOX with published weights generated by standard image datasets. We also propose a new regression strategy for 24-points object representation method, which is insensitive to image distortion. The experiments show that the feature extraction part is robust to fisheye image features, while the regression part of location and category performs poorly. The strategy can achieve the position of discrete points without calculating the IOU of irregular-shaped boxes. Theoretically, the strategy can be widely adopted to regress the irregular bounding boxes composed of discrete points. Source code is at https://github.com/IN2-ViAUn/Exploration-of-Potential.