For an autonomous surface vessel (ASV) to dock, it must track other vessels close to the docking area. Kayaks present a particular challenge due to their proximity to the dock and relatively small size. Maritime target tracking has typically employed land masking to filter out land and the dock. However, imprecise land masking makes it difficult to track close-to-dock objects. Our approach uses Light Detection And Ranging (LiDAR) data and maps the docking area before tracking. The precise 3D measurements allow for precise map creation. However, the mapping could result in static, yet potentially moving, objects being mapped. We detect and filter out potentially moving objects from the LiDAR data by utilizing image data. The visual vessel detection and segmentation method is a neural network that is trained on our labeled data. Close-to-shore tracking improves with an accurate map and is demonstrated on a recently gathered real-world dataset. The dataset contains multiple sequences of a kayak and a day cruiser moving close to the dock, in a collision path with an autonomous ferry prototype.

Unmanned surface vehicles (USVs) require robust situational awareness to navigate safely in complex maritime environments. A critical element of this is to identify the free navigable space around the USV. Free water regions can be derived from water segmentation in the image. However, these segmented regions must be transformed into a bird's eye view (BEV) representation to be utilized effectively in motion planning. This paper proposes a novel approach to estimate free navigable space in a BEV format by integrating a stereo camera and light detection and ranging (LiDAR). The proposed method uses water segmentation to delineate the water surface and represents the closest obstacles in the USV line of sight using vertical planar rectangles known as Stixels. The depth of these Stixels is derived from LiDAR data, ensuring precise positioning in space. The effectiveness of the approach is demonstrated through experiments conducted on real-world data collected from the milliAmpere 2 (MA2) autonomous ferry prototype in Trondheim, Norway. Qualitative evaluations focusing on accuracy and temporal consistency confirm its ability to reliably detect free navigable areas in complex maritime environments.

Target tracking is essential for autonomous vehicles to avoid collisions. Using a stereo camera for the target tracking gives a dense representation of the targets, contrary to the the sparser data on typical radars and lidars. With a wider baseline stereo camera the depth measurements are more accurate, but the stereo matching challenge is greater, especially in the maritime domain with reflections on the water. Earlier classical methods of tracking using stereo cameras have often tracked targets by first doing water surface estimation and then finding objects perturbing the plane. The challenge is then to get a good estimate of the water surface plane while still having precise measurements to the targets. We propose both a short baseline method and a multi-baseline method for target detection. The multi-baseline method uses a short baseline stereo camera to find the water plane and uses a wider baseline stereo camera to get accurate target measurements. The targets are consistently being tracked when using data collected during the summer of 2023 from an autonomous ferry prototype compared to ground truth GNSS tracks. The short baseline method achieves minimal error for a day cruiser boat 40 m away using a camera baseline of only 12 cm. The multi-baseline method further improves the accuracy of boat measurements, especially for a far-away small kayak.

Navigating unmanned surface vehicles (USVs) in urban waterways presents unique challenges due to irregular waterlines, obstacles, and reflections in the water. Determining the collision-free navigable area is crucial to enable safe USV operation. This paper introduces Fused Water Surface Segmentation (FusedWSS), a novel approach to water surface segmentation that aims to enhance navigation capabilities for USVs in complex harbor environments using a stereo camera. The method locates the water plane by performing plane fitting with outlier rejection and plane validation on the reconstructed 3D point cloud. From the plane parameters, the virtual horizon line is inferred and used for point cloud and image cropping. The water surface mask and virtual horizon line are fused with a deep learningbased semantic segmentation method to produce accurate and reliable water masks for each image frame. Additional refinement of the water mask is performed using detected obstacle masks. Validation was carried out using data from the MilliAmpere 2 autonomous ferry prototype in Trondheim, Norway, and a publicly available maritime dataset, demonstrating the efficacy of the methods.

The momentum towards autonomous technology is building up in the maritime domain, as the automotive industry has made big steps towards autonomous driving. The automotive industry has increasingly utilized visual methods for multi-object tracking (MOT), with the help of accessible benchmarking datasets such as KITTI. This paper presents a tracking pipeline that tracks in the world frame by using elements of a well-established visual tracking method that tracks objects in the image frame. The pipeline fuses 3D information from lidar or stereo vision with object masks from a deep learning-based ship detector. To handle occlusions, we implemented a track manager that predicts lost objects’ movement until they reappear. Also, we provide a comparison between using lidar and stereo as the depth modality in the tracking pipeline. Results from a real-world experiment indicate that camera-lidar fusion gives consistently precise estimates, while the precision with stereo depends on the range and the type of vessel tracked.

In autonomous vehicle systems, it is paramount to detect other objects in the vicinity and track their movement. Extended Object Tracking (EOT) provides a convenient framework for tracking objects using high-resolution sensor data by defining models for the object’s spatial dimensions (a.k.a. extent). In maritime applications, the objects of interest are mainly other maritime vessels, and these vary greatly in shape and size. This diversity proves to be a challenge for defining general extent models that both give accurate representations for most vessels and that do not depend on a large number of parameters. In this paper, a general three-dimensional low-parameter ship hull model designed for EOT is presented. The presented extent model is constructed by intertwining a polynomial representation along the vertical direction with a frequency representation along the horizontal plane. However, to reduce the dimension of the parameter space without compromising its accuracy, the horizontal frequency representation is modified by performing a Principal Component Analysis (PCA). In particular, this extent representation does not require an underlying discretization grid, which makes the model scalable and therefore well-suited for modeling objects that vary greatly in size.

Future autonomous ships will need several redundant positioning systems to navigate reliably. Global Navigation Satellite Systems are highly accurate but they are susceptible to disruptions and intentional jamming. Maritime radars have long range and are robust against bad weather and darkness, but the use for ownship motion estimation has received relatively little attention in the research field. In this work, we present a radar odometry estimation method inspired by advances in visual odometry and simultaneous localization and mapping. The method works on raw radar data in a coastal environment and combines the Kanade-Lucas-Tomashi tracker with a factor graph back-end. We test it on data from a large ship with a maritime radar with a range of 19 km. We find that it is robust with only a small drift and no erroneous jumps in the estimate.

Extended Object Tracking (EOT) is a advantageous technique for achieving situational awareness in autonomous vehicle systems. The EOT problem is to both estimate the movement and spatial dimensions of an object using high-resolution measurements. In the case of laser measurements or other types of measurements that correspond to points on the object’s boundary, the true measurement model of the EOT problem is based on an implicit equation for the measurement coordinates. This intrinsic implicity is often not addressed directly in several EOT models found in the literature. In this paper, the EOT problem is reformulated as a least square minimization problem without compromising the original implicit measurement model by introducing an extra variable for each measurement. In addition, this new least squares formulation allows considering measurements and state variables for a whole time window, and not just a single time step. An EOT algorithm based on solving the derived least squares minimization problem is proposed and tested with simulated scenarios.