In many applications of extended object tracking, 2-dimensional models are used due to the simplicity of the parametrization of the extent compared to a 3-dimensional model. However, many high-resolution sensors today provide a 3D point cloud which represents a challenge for how to relate these measurements with a 2D model. This is particularly relevant when modeling the extent using a parametrization of its contour, as with the random hypersurface model, since the measurements need to be related to the boundary of the object, whereas the measurements may also be generated from the interior of the object. The random hypersurface model uses a scaling factor to account for measurements being generated across the surface defined by the contour. In this article, we use the Gaussian process model and explore different ways to handle interior measurements either by using a boundary model, a surface model, or a combination of the two and we present a study on both simulated and real maritime data of these different approaches. The results show that using the boundary model results in a more precise extent estimate but one which could be underestimated, whereas incorporating the surface model results in a more conservative, overestimated extent. We also show that estimating the scaling factor for the surface model generally results in improved performance.

For an autonomous surface vessel (ASV) to dock, it must track other vessels close to the docking area. Kayaks present a particular challenge due to their proximity to the dock and relatively small size. Maritime target tracking has typically employed land masking to filter out land and the dock. However, imprecise land masking makes it difficult to track close-to-dock objects. Our approach uses Light Detection And Ranging (LiDAR) data and maps the docking area before tracking. The precise 3D measurements allow for precise map creation. However, the mapping could result in static, yet potentially moving, objects being mapped. We detect and filter out potentially moving objects from the LiDAR data by utilizing image data. The visual vessel detection and segmentation method is a neural network that is trained on our labeled data. Close-to-shore tracking improves with an accurate map and is demonstrated on a recently gathered real-world dataset. The dataset contains multiple sequences of a kayak and a day cruiser moving close to the dock, in a collision path with an autonomous ferry prototype.

Unmanned surface vehicles (USVs) require robust situational awareness to navigate safely in complex maritime environments. A critical element of this is to identify the free navigable space around the USV. Free water regions can be derived from water segmentation in the image. However, these segmented regions must be transformed into a bird's eye view (BEV) representation to be utilized effectively in motion planning. This paper proposes a novel approach to estimate free navigable space in a BEV format by integrating a stereo camera and light detection and ranging (LiDAR). The proposed method uses water segmentation to delineate the water surface and represents the closest obstacles in the USV line of sight using vertical planar rectangles known as Stixels. The depth of these Stixels is derived from LiDAR data, ensuring precise positioning in space. The effectiveness of the approach is demonstrated through experiments conducted on real-world data collected from the milliAmpere 2 (MA2) autonomous ferry prototype in Trondheim, Norway. Qualitative evaluations focusing on accuracy and temporal consistency confirm its ability to reliably detect free navigable areas in complex maritime environments.

In maritime situational awareness, the Automatic Identification System (AIS) is a vital source of information. Recent work has explored the fusion of AIS information and exteroceptive measurements to improve maritime target tracking performance, also for extended object tracking. However, in extended object tracking, the discrepancy between the center of the ship and the position reported by the AIS system is no longer negligible and is a source of systemic bias, which can degrade tracking performance. In this paper, we introduce a method for estimating this discrepancy based on AIS information and the estimation provided by the Gaussian process target model from the exteroceptive sensor data. We use this method combined with an extended object Poisson multi-Bernoulli mixture (PMBM) filter to perform multiple extended object tracking. We also introduce a specific method for initialization of targets using AIS measurements in this filter. We validate the proposed method with LiDAR and AIS data, collected from an inland waterway in Belgium. The results show that compensating for the bias in this manner results in better tracking performance, primarily due to better initialization of new targets.

Target tracking is essential for autonomous vehicles to avoid collisions. Using a stereo camera for the target tracking gives a dense representation of the targets, contrary to the the sparser data on typical radars and lidars. With a wider baseline stereo camera the depth measurements are more accurate, but the stereo matching challenge is greater, especially in the maritime domain with reflections on the water. Earlier classical methods of tracking using stereo cameras have often tracked targets by first doing water surface estimation and then finding objects perturbing the plane. The challenge is then to get a good estimate of the water surface plane while still having precise measurements to the targets. We propose both a short baseline method and a multi-baseline method for target detection. The multi-baseline method uses a short baseline stereo camera to find the water plane and uses a wider baseline stereo camera to get accurate target measurements. The targets are consistently being tracked when using data collected during the summer of 2023 from an autonomous ferry prototype compared to ground truth GNSS tracks. The short baseline method achieves minimal error for a day cruiser boat 40 m away using a camera baseline of only 12 cm. The multi-baseline method further improves the accuracy of boat measurements, especially for a far-away small kayak.

Navigating unmanned surface vehicles (USVs) in urban waterways presents unique challenges due to irregular waterlines, obstacles, and reflections in the water. Determining the collision-free navigable area is crucial to enable safe USV operation. This paper introduces Fused Water Surface Segmentation (FusedWSS), a novel approach to water surface segmentation that aims to enhance navigation capabilities for USVs in complex harbor environments using a stereo camera. The method locates the water plane by performing plane fitting with outlier rejection and plane validation on the reconstructed 3D point cloud. From the plane parameters, the virtual horizon line is inferred and used for point cloud and image cropping. The water surface mask and virtual horizon line are fused with a deep learningbased semantic segmentation method to produce accurate and reliable water masks for each image frame. Additional refinement of the water mask is performed using detected obstacle masks. Validation was carried out using data from the MilliAmpere 2 autonomous ferry prototype in Trondheim, Norway, and a publicly available maritime dataset, demonstrating the efficacy of the methods.

In the automotive community, methods for tracking, localization,and situational awareness are routinely tested on well-known open-source datasets from the real world. In many other applications of target tracking, such as maritime radar tracking, there is a lack of such data. In this paper, we present a large dataset consisting of data recorded by a frequency-modulated continuous wave radar overlooking the Trondheim City Canal over several weeks during the summer of 2023. The dataset includes a rich variety of boat traffic, ranging from large ferries to formations of kayaks. All the data have been analyzed by means of classical joint integrated probabilistic data association-based multiple target tracking. We point out several challenges that arise in this dataset, such as merged measurements and multipath. We also demonstrate that the data are sufficient to generate statistical information about traffic patterns in the City Canal.

The detection of small and dim targets under low signal-to-noise ratio (SNR) circumstances is a commonly encountered yet challenging endeavour in radar signal processing. The standard approach to deal with undesirable background conditions involves coherent processing and integration with subsequent detection directly applied to the radar signals. However, optical flow, a widespread visual tracking method, has rarely been used in this context. In this paper, we address the issue of radar target detection in low SNR scenarios by employing optical flow on radar images. This work focuses on the divergence of the optical flow vector field, utilising a novel approach of coherently integrating consecutive flow fields calculated against a homogeneous reference plane. The proposed methodology allows for more robust target identification and thus a precise initialisation of tracking systems. To validate and demonstrate the benefits of the proposed approach, simulations are conducted and discussed.

The momentum towards autonomous technology is building up in the maritime domain, as the automotive industry has made big steps towards autonomous driving. The automotive industry has increasingly utilized visual methods for multi-object tracking (MOT), with the help of accessible benchmarking datasets such as KITTI. This paper presents a tracking pipeline that tracks in the world frame by using elements of a well-established visual tracking method that tracks objects in the image frame. The pipeline fuses 3D information from lidar or stereo vision with object masks from a deep learning-based ship detector. To handle occlusions, we implemented a track manager that predicts lost objects’ movement until they reappear. Also, we provide a comparison between using lidar and stereo as the depth modality in the tracking pipeline. Results from a real-world experiment indicate that camera-lidar fusion gives consistently precise estimates, while the precision with stereo depends on the range and the type of vessel tracked.

In autonomous vehicle systems, it is paramount to detect other objects in the vicinity and track their movement. Extended Object Tracking (EOT) provides a convenient framework for tracking objects using high-resolution sensor data by defining models for the object’s spatial dimensions (a.k.a. extent). In maritime applications, the objects of interest are mainly other maritime vessels, and these vary greatly in shape and size. This diversity proves to be a challenge for defining general extent models that both give accurate representations for most vessels and that do not depend on a large number of parameters. In this paper, a general three-dimensional low-parameter ship hull model designed for EOT is presented. The presented extent model is constructed by intertwining a polynomial representation along the vertical direction with a frequency representation along the horizontal plane. However, to reduce the dimension of the parameter space without compromising its accuracy, the horizontal frequency representation is modified by performing a Principal Component Analysis (PCA). In particular, this extent representation does not require an underlying discretization grid, which makes the model scalable and therefore well-suited for modeling objects that vary greatly in size.

In Multiple Extended Object Tracking, the PMBM (Poisson Multi-Bernoulli Mixture) tracker is considered state-of-the-art. Originally, it was presented with the GGIW (Gamma Gaussian Inverse Wishart) target model, which is a random matrix model. When tracking larger objects using LiDAR, measurements are generated by the contour rather than the whole target surface, and it is beneficial to model this with the target model. A target model which has this capability is the Gaussian Process (GP) extent model. This paper presents a PMBM tracker using this target model. We also discuss considerations related to the use of the GP model in the PMBM framework. Secondly, we present improvements in the target model which increases the robustness of the model by dealing with the inherent nonlinearities using the Gauss-Newton method. We also present a comparison with the GGIW-PMBM tracker on simulated and real LiDAR data gathered from maritime vessels.

By developing a method for automatically calibrating the extrinsic parameters of ship-mounted cameras, this paper tests combining Structure from Motion-algorithms with Hand-Eye calibration solvers in a novel algorithm which demonstrates an ability to discern the orientation of cameras with accuracy comparable to- or better than current manual methods, proven through tests with both synthetic and real-world data.

Future autonomous ships will need several redundant positioning systems to navigate reliably. Global Navigation Satellite Systems are highly accurate but they are susceptible to disruptions and intentional jamming. Maritime radars have long range and are robust against bad weather and darkness, but the use for ownship motion estimation has received relatively little attention in the research field. In this work, we present a radar odometry estimation method inspired by advances in visual odometry and simultaneous localization and mapping. The method works on raw radar data in a coastal environment and combines the Kanade-Lucas-Tomashi tracker with a factor graph back-end. We test it on data from a large ship with a maritime radar with a range of 19 km. We find that it is robust with only a small drift and no erroneous jumps in the estimate.

We present a novel target tracking algorithm, which is designed to track a target in the presence of wake clutter. What distinguishes the method from previous wake-compensating trackers is that it also models the existence of the target and exploits the information provided by the wake measurements for this purpose. We present two ways of modeling the wake, and we evaluate the algorithm’s performance on simulated data. Results show that the method improves upon comparable target tracking methods when wake measurements are present.

Extended Object Tracking (EOT) is a advantageous technique for achieving situational awareness in autonomous vehicle systems. The EOT problem is to both estimate the movement and spatial dimensions of an object using high-resolution measurements. In the case of laser measurements or other types of measurements that correspond to points on the object’s boundary, the true measurement model of the EOT problem is based on an implicit equation for the measurement coordinates. This intrinsic implicity is often not addressed directly in several EOT models found in the literature. In this paper, the EOT problem is reformulated as a least square minimization problem without compromising the original implicit measurement model by introducing an extra variable for each measurement. In addition, this new least squares formulation allows considering measurements and state variables for a whole time window, and not just a single time step. An EOT algorithm based on solving the derived least squares minimization problem is proposed and tested with simulated scenarios.

This paper explores evaluation of association marginals in multiple hypothesis tracking. The work builds upon recent results where loop belief propagation (LBP) has been used in single-hypothesis cases. There are two contributions in the paper. The first is a novel factor graph representation of the joint multi-hypothesis association posterior. The second contribution is two algorithms that both use LBP to evaluate association marginals. The first method uses total probability in conjunction with hypothesis-conditioned LBP, and is called PHD-LBP. The second method is an LBP algorithm running directly on the full multi-hypothesis association graph with novel, specialized message definitions that are derived in this paper and efficient to compute and store in memory, and is called MH-LBP. Results show that both algorithms perform well with high correlation with the exact marginals for the majority of the cases.

Track initiation in multi-object tracking for groups of objects traveling close to each other may require considering many unlikely tracks near each other. Limiting track numbers by not initiating within the gate, as is commonly done, does not work well in these scenarios. With many tracks, computing the exact marginal track to measurement probabilities in a joint integrated probabilistic data association (JIPDA) is computationally expensive.In addressing this problem, we consider the linear multitarget (LM) IPDA, which is a linear approximation of the data association in JIPDA. Here, we formulate it as a Poisson point process (PPP) approximation of the track measurement densities with a particular intensity function. Given our focus on track initialization, we devise how to use the LM approximation on only a subset of the tracks, while the other tracks can be treated as in JIPDA after that. This gives a novel new approximation of the track to measurement data association probabilities which we term LMS.Simulations show that using the LM technique is superior to the PHD filter in terms of posterior track existence probability. Further, it is seen that the PHD as an intensity function in LM performs worse than the original LM in terms of the data association probabilities. The LMS data association probabilities are also shown to typically have better worst-case errors than the original LM and loopy belief propagation (LBP). In terms of the GOSPA metric, initiating tracks on every measurement using LMS gives more reliable and faster track initialization for objects appearing close to already established tracks.

This paper proposes a pipeline for feature-based laser odometry for autonomous surface vehicles (ASVs) operating in urban environments. In particular, we investigate the suitability of several keypoint extractors and keypoint descriptors available through the Point Cloud Library (PCL). The complete odometry system, using the different extractors and descriptors, is implemented using the iSAM2 framework, and validated on real lidar data recorded onboard the autonomous ferry prototype MilliAmpere. The results demonstrate that accuracy similar to a standard Global Navigation Satellite System (GNSS) receiver can be achieved after 10 minutes even without loop closure. This study can be used as a starting point for future research on Simultaneous Localization And Mapping (SLAM) for ASVs.

A robust and efficient Collision Avoidance (COLAV) system for autonomous ships is dependent on a high degree of situational awareness. This includes inference of the intent of nearby obstacles, including compliance with traffic rules such as COLREGS, in order to enable more intelligent decision making for the autonomous agent. Here, a generalized framework for obstacle intent inference is introduced. Different obstacle intentions are then considered in the Probabilistic Scenario-Based Model Predictive Control (PSB-MPC) COLAV algorithm using an examplatory intent model, when statistics about traffic rules compliance and the next waypoint for an obstacle are assumed known. Simulation results show that the resulting COLAV system is able to make safer decisions when utilizing the extra intent information.

In multiple hypothesis tracking (MHT) the outcome space is partitioned into a discrete collection of events known as association hypotheses, whose posterior probabilities are calculated. The discrete nature of this problem means that it can be viewed as a classification problem. In this paper we argue that the hypothesis probabilities must obey some bounds from classification theory. These bounds can be used to investigate the correctness of MHT implementations.

When operating an autonomous surface vessel (ASV) in a marine environment it is vital that the vessel is equipped with a collision avoidance (COLAV) system. This system must be able to predict the trajectories of other vessels in order to avoid them. The increasingly available automatic identification system (AIS) data can be used for this task. In this paper, we present a data-driven approach to predict vessel positions 5–15 minutes into the future using AIS data. The predictions are given as Gaussian Mixture Models (GMMs), thus the predictions give a measure of uncertainty and can handle multimodality. A nearest neighbor algorithm is applied on two different data structures. Tests to determine the accuracy and covariance consistency of both structures are performed on real data.

Reliable track initiation is an important component of a tracking system, especially when it is used as part of a more general collision avoidance (COLAV) system. Some tracking methods (e.g., IPDA) come with an in-built track initiation capability, while other methods (e.g., JPDA) lack this capability, which in such cases typically is taken care of by heuristic rules such as the M/N logic. Although Reid's multiple hypothesis tracker (MHT) is capable of track initiation, many implementations do not include track initiation in the MHT framework due to the increased complexity. While MHT is fundamentally Bayesian, the non-Bayesian sequential probability ratio test (SPRT) of Van Keuk is often used for track initiation. In this paper we derive a Bayesian SPRT for track initiation based on Reid's MHT. The approach is compared with the classical SPRT, both from a theoretical perspective and using simulations. Furthermore, the paper provides a comparison between the two SPRT versions, the IPDA and the M/N logic in terms of system operating characteristic (SOC) curves and track initiation time. The initiation methods are also tested on real radar data recorded during full-scale maritime COLAV experiments.

The Simultaneous Localization And Mapping (SLAM) estimation problem is a nonlinear problem, due to the nature of the range and bearing measurements. In latter years it has been demonstrated that if the nonlinearities from the attitude are handled by a separate nonlinear observer, the SLAM dynamics can be represented as a linear time varying (LTV) system, by introducing these nonlinearities and nonlinear measurements as time varying vectors and matrices. This makes the SLAM estimation problem globally solvable with a Kalman filter, however, the noise structure is no longer trivial. In this paper, a new bearings only SLAM estimation algorithm is presented, including a novel design of the noise covariance matrices. Simulations of the SLAM estimator are presented, and show the performance of the state and uncertainty estimates, as well as the stability of the proposed estimator.

The multiple hypothesis tracker (MHT) has historically been considered a gold standard for multi-target tracking. In this paper we show that the key formula for hypothesis probabilities in Reid's MHT can be derived from the modern theory of finite set statistics (FISST) insofar as appropriate assumptions (Poisson models for clutter and undetected targets, no target-death, linear-Gaussian Markov target kinematics) are adhered to.

In order for autonomous surface vessels (ASVs) to avoid collisions at sea it is necessary to predict the future trajectories of surrounding vessels. This paper investigate the use of historical automatic identification system (AIS) data to predict such trajectories. The availability of AIS data have steadily increased in the last years as a result of more regulations, together with wider coverage through AIS integration on satellites and more land based receivers. Several AIS-based methods for predicting vessel trajectories already exist. However, these prediction techniques tend to focus on time horizons in the level of hours. The prediction time of our interest typically ranges from a few minutes up to about 15 minutes, depending on the maneuverability of the ASV. This paper presents a novel datadriven approach which recursively use historical AIS data in the neighborhood of a predicted position to predict next position and time. Three course and speed prediction methods are compared for one time step predictions. Lastly, the algorithm is briefly tested for multiple time steps in curved environments and shows good potential.