Maritime environments often present hazardous situations due to factors such as moving ships or buoys, which become obstacles under the influence of waves. In such challenging conditions, the ability to detect and track potentially hazardous objects is critical for the safe navigation of marine robots, but datasets capturing these scenarios remain limited. To address this limitation, we introduce a new multi-modal dataset that includes image and point-wise annotations of maritime obstacles. Our dataset provides detailed ground truth for obstacle detection and tracking, including objects as small as 10 × 10 pixels, which are crucial for maritime safety. To validate the dataset's effectiveness as a reliable benchmark, we conducted evaluations using various methodologies, including state-of-the-art (SOTA) techniques for object detection and tracking. These evaluations are expected to contribute to improving performance, particularly in the complex maritime environment. This represents the first demonstration of a dataset offering multi-modal annotations specifically tailored to maritime environments. Our dataset is available at https://github.com/sparolab/PoLaRIS.