Indiscernible marine object counting refers to the counting of marine objects that are visually blended with their surrounding environment. This task encounters critical challenges, including limited visibility in underwater scenes, mutual occlusion and overlap among objects, and the dynamic similarity in appearance, color, and texture between the background and foreground. To address the scarcity of video-based indiscernible object counting datasets, we have established a new dataset comprising 50 videos, from which approximately 800 frames have been extracted and annotated with around 40, 800 point-wise object labels. This dataset represents real underwater environments where indiscernible marine objects are intricately integrated with their surroundings, thereby comprehensively illustrating the aforementioned challenges in marine object counting. To address these challenges, we propose a depth-assisted network with adaptive motion-differentiated feature encoding. The network consists of a backbone encoding module and three branches: a depth-assisting branch, a density estimation branch, and a motion weight generation branch. Depth-aware features extracted by the depth-assisting branch are enhanced via a depth-enhanced encoder to improve object representation. Meanwhile, weights from the motion weight generation branch refine multi-scale perception features in the adaptive flow estimation module. Experiments demonstrate that our method not only achieves state-of-the-art performance on the proposed dataset but also yields competitive results on three video-based crowd counting datasets. The pre-trained model, code, and dataset are publicly available at https://github.com/OUCVisionGroup/VIMOC-Net.

Underwater image quality is seriously degraded due to the insufficient light in water. Although artificial illumination can assist imaging, it often brings non-uniform illumination phenomenon. To this end, we develop an illumination channel sparsity prior (ICSP) guided variational framework for non-uniform illumination underwater image restoration. Technically, the illumination channel sparsity prior is built on the observation that the illumination channel of a uniform-light underwater image in HSI color space contains few pixels whose intensity is very low. Then according to the Retinex theory, we design a variational model with L0 norm term, constraint term, and gradient term, by integrating the proposed ICSP into an extended underwater image formation model. Such three regularizations are effective in enhancing the brightness, correcting color distortion, and revealing structures and fine-scale details. Meanwhile, we exploit a fast numerical algorithm on the base of the alternating direction method of multipliers (ADMM) to accelerate solving this optimization problem. We also collect a benchmark dataset, namely NUID that contains 925 real underwater images of different non-uniform illumination. Extensive experiments demonstrate that our proposed method is effective in terms of qualitative and quantitative comparisons, ablation studies, convergence analysis, and applications. The code and dataset are available athttps://github.com/Hou-Guojia/ICSP.

Vision is an important source of information for underwater observations, but underwater images commonly suffer severe visual degradation due to the complexity of the underwater imaging environment and wavelength-dependent absorption effects. There is an urgent need for underwater image enhancement techniques to improve the visual quality of underwater images. Due to the scarcity of high-quality paired training samples, underwater image enhancement based on deep learning has never achieved success similar to other vision tasks. Instead of learning complicated distortion-to-clear mappings with deep networks, we design a template-free color transfer learning framework for predicting transfer parameters, which are more easily captured and described. In addition, we add attention-driven modules to learn differentiated transfer parameters for more flexible and robust enhancement. We verify the effectiveness of our method on multiple publicly available datasets and show its efficiency in enhancing high-resolution images. The source code and the trained models are available on the project homepage: https://trentqq.github.io/TCTL-Net.html.

Underwater images suffer from quality degradation due to the underwater light absorption and scattering. It remains challenging to enhance underwater images using deep learning-based methods since the scarcity of real-world underwater images and their enhanced counterparts. Although existing works manually select well-enhanced images as reference images to train enhancement networks in an end-to-end manner, their performance tends to be inferior in some scenarios. We argue that the manually selected reference images cannot approximate their ground truth perfectly, leading to imbalanced learning and domain shift in enhancement networks. To address this issue, we analyse widely used underwater datasets from the perspective of color spectrum distribution and surprisingly find the sound color spectrum distribution of the enhanced reference images compared to in-air datasets. Based on this perceptive observation, instead of directly learning the enhancement mapping, we propose a novel methodology to learn color compensation for general purposes. Specifically, we present a probabilistic color compensation network that estimates the probabilistic distribution of colors by multi-scale volumetric fusion of texture and color features. We further propose a novel two-stage enhancement framework that first performs color compensation and then enhancement, which is highly flexible to be integrated with an existing enhancement method without tuning. Extensive experiments on underwater image enhancement across various challenging scenarios show that our proposed approach consistently improves the results of the popular conventional and learning-based methods by a significant margin. Moreover, our enhanced images achieve superior performance on underwater salient object detection and visual 3D reconstruction, demonstrating that our method can successfully break through the generalization bottleneck of existing learning-based enhancement models. Our implementation will be made available at https://github.com/Ray2OUC/P2CNet.

Estimating depth from a single underwater image is one of the main tasks of underwater visual perception. However, data-driven underwater depth estimation methods have long been challenging to make breakthroughs due to the difficulty of obtaining a large number of true-value references. This is partly due to the high cost of acquisition equipment, which is difficult to be applied to diverse ocean scenes by a wide range of users, and therefore sample diversity is difficult to guarantee; on the other hand, manual annotation of dense depth relationships is almost impossible to achieve. In this paper, we establish a new underwater relative depth estimation benchmark, namely SUIM-SDA, by extending the SUIM dataset with more than 6,000 manually annotated depth trendlines, 25 million pixels with paired depth-ranking labels and 14 million depth-ranked pixel pairs. Using the sparse depth relation annotation provided by SUIM-SDA and the semantic information provided by SUIM, we design a new multi-stage multi-task learning framework to predict a dense relative depth map for a single underwater image. Comprehensive comparison and ablation study on the publicly available dataset and our new benchmark demonstrate the effectiveness of the proposed weakly-supervised strategy for dense relative depth estimation. The new benchmark, source code, and trained models are available on the project home page: https://wangxy97.github.io/WsUIDNet.

Depth completion aims to recover pixelwise depth from incomplete and noisy depth measurements with or without the guidance of a reference RGB image. This task attracted considerable research interest due to its importance in various computer vision-based applications, such as scene understanding, autonomous driving, 3-D reconstruction, object detection, pose estimation, trajectory prediction, and so on. As the system input, an incomplete depth map is usually generated by projecting the 3-D points collected by ranging sensors, such as LiDAR in outdoor environments, or obtained directly from RGB-D cameras in indoor areas. However, even if a high-end LiDAR is employed, the obtained depth maps are still very sparse and noisy, especially in the regions near the object boundaries, which makes the depth completion task a challenging problem. To address this issue, a few years ago, conventional image processing-based techniques were employed to fill the holes and remove the noise from the relatively dense depth maps obtained by RGB-D cameras, while deep learning-based methods have recently become increasingly popular and inspiring results have been achieved, especially for the challenging situation of LiDAR-image-based depth completion. This article systematically reviews and summarizes the works related to the topic of depth completion in terms of input modalities, data fusion strategies, loss functions, and experimental settings, especially for the key techniques proposed in deep learning-based multiple input methods. On this basis, we conclude by presenting the current status of depth completion and discussing several prospects for its future research directions.

Finding correspondences between a pair of images is the key ingredient for many applications such as localization and panorama. However, due to a variety of challenges between multi-view images in practice, the results of using a single kind of descriptor may vary significantly across different scenes. In this paper, we treat the assignment task as a clustering problem and propose an image matching method that fuses multiple descriptors to tackle the above difficulties. First, we extract multiple descriptors at the keypoints on two images. Then, we compute a pairwise similarity matrix for each kind of descriptor. Afterwards, we compute a weighted combination of these similarity matrices, and use it to build correspondences via a modified multi-kernel clustering module. The proposed method is tested on three public image datasets: two ground image sets and an Unmanned Aerial Vehicle (UAV) image set. Experiments show that the proposed method can adapt to different number of descriptors. It significantly improves the matching accuracy in a variety of scenarios and downstream tasks.

Due to the wavelength-dependent light absorption and scattering, the raw underwater images are usually inevitably degraded. Underwater image enhancement (UIE) is of great importance for underwater observation and operation. Data-driven methods, such as deep learning-based UIE approaches, tend to be more applicable to real underwater scenarios. However, the training of deep models is limited by the extreme scarcity of underwater images with enhancement references, resulting in their poor performance in dynamic and diverse underwater scenes. As an alternative, enhancement reference achieved by volunteer voting alleviate the sample shortage to some extent. Since such artificially acquired references are not veritable ground truth, they are far from complete and accurate to provide correct and rich supervision for the enhancement model training. Beyond training with single reference, we propose the first comparative learning framework for UIE problem, namely CLUIE-Net, to learn from multiple candidates of enhancement reference. This new strategy also supports semi-supervised learning mode. Besides, we propose a regional quality-superiority discriminative network (RQSD-Net) as an embedded quality discriminator for the CLUIE-Net. Comprehensive experiments demonstrate the effectiveness of RQSD-Net and the comparative learning strategy for UIE problem. The code, models and new dataset RQSD-UI are available at: https://justwj.github.io/CLUIE-Net.html/.

Achieving subjective and objective quality assessment of underwater images is of high significance in underwater visual perception and image/video processing. However, the development of underwater image quality assessment (UIQA) is limited for the lack of publicly available underwater image datasets with human subjective scores and reliable objective UIQA metrics. To address this issue, we establish a large-scale underwater image dataset, dubbed UID2021, for evaluating no-reference (NR) UIQA metrics. The constructed dataset contains 60 multiply degraded underwater images collected from various sources, covering six common underwater scenes (i.e., bluish scene, blue-green scene, greenish scene, hazy scene, low-light scene, and turbid scene), and their corresponding 900 quality improved versions are generated by employing 15 state-of-the-art underwater image enhancement and restoration algorithms. Mean opinion scores with 52 observers for each image of UID2021 are also obtained by using the pairwise comparison sorting method. Both in-air and underwater-specific NR IQA algorithms are tested on our constructed dataset to fairly compare their performance and analyze their strengths and weaknesses. Our proposed UID2021 dataset enables ones to evaluate NR UIQA algorithms comprehensively and paves the way for further research on UIQA. The dataset is available at https://github.com/Hou-Guojia/UID2021 .

Abstract Due to the inevitable wavelength‐dependent light absorption and forward/backward scattering, underwater images usually suffer severe color distortion and are hazy. It has become quite necessary to improve the visual quality of underwater images for both underwater observation and operation. Traditional enhancement methods and existing deep learning‐based approaches to underwater image enhancement usually produce unsatisfactory results for photographs taken in complicated, wild underwater scenes. In such scenes, complex and diverse degradation‐enhancement mappings are often difficult to model, especially since there are very limited samples available for learning. Inspired by the success of color‐transfer techniques, it is found that clear template image‐assisted color transfer is a promising strategy for underwater image enhancement, including not only color correction but also contrast and visibility improvement. Therefore, instead of directly learning the complex deep enhancement models, it is proposed to select proper color‐transfer templates by learning the latent consistency between the templates and the raw underwater images. The proposed new enhancement strategy alleviates the problem caused by incomplete color‐correction models and provides more stable enhancements by utilizing color transfer with consideration of global color distribution consistency and local visual contrast. Comprehensive experiments conducted on UIEB, RUIE, URPC and SQUID datasets demonstrate the good performance and great potential of the proposed new underwater image enhancement strategy.

In underwater scenes, degraded underwater images caused by wavelength-dependent light absorption and scattering present huge challenges to vision tasks. Underwater image enhancement has attracted much attention due to the significance of vision-based applications in marine engineering and underwater robotics. Numerous underwater image enhancement algorithms have been proposed in the last few years. However, almost all existing approaches focus only on the enhancement of independent images. Considering that images photographed in the same underwater scene usually share similar degradation, related images can provide rich complementary information for each other’s enhancement. In this paper, we propose an Underwater Image Co-enhancement Network (UICoE-Net) based on an encoder-decoder Siamese architecture. For joint learning, we introduced correlation feature matching units into the multiple layers of our Siamese encoder-decoder structure in order to communicate the mutual correlation of the two branches. Extensive experiments using the Underwater Image Enhancement Benchmark (UIEB), Underwater Image Co-enhancement Dataset (UICoD) collected from an underwater video dataset with ground-truth reference and Stereo Quantitative Underwater Image Dataset (SQUID) dataset demonstrate the effectiveness of our method.

Due to the wavelength-dependent light attenuation, refraction and scattering, underwater images usually suffer from color distortion and blurred details. However, due to the limited number of paired underwater images with undistorted images as reference, training deep enhancement models for diverse degradation types is quite difficult. To boost the performance of data-driven approaches, it is essential to establish more effective learning mechanisms that mine richer supervised information from limited training sample resources. In this paper, we propose a novel underwater image enhancement network, called SGUIE-Net, in which we introduce semantic information as high-level guidance via region-wise enhancement feature learning. Accordingly, we propose semantic region-wise enhancement module to better learn local enhancement features for semantic regions with multi-scale perception. After using them as complementary features and feeding them to the main branch, which extracts the global enhancement features on the original image scale, the fused features bring semantically consistent and visually superior enhancements. Extensive experiments on the publicly available datasets and our proposed dataset demonstrate the impressive performance of SGUIE-Net. The code and proposed dataset are available at https://trentqq.github.io/SGUIE-Net.html.

In this letter, we present a novel rotation averaging pipeline, which is performed in a hierarchical manner. Unlike the traditional rotation averaging methods which focus on designing robust loss function to get rid of the impacts of the relative rotation outliers, here the outliers are detected and filtered by leveraging the well-known robust model estimation procedure, RANdom SAmple Consensus (RANSAC). During the RANSAC process, the minimal set is randomly sampled by random tree spanning on the Epipolar-geometry Graph (EG). As the RANSAC estimation result is sensitive to the size of minimal set, the EG is clustered into several sub-graphs, and the inner- and inter-cluster RANSAC-based rotation averaging are performed hierarchically. In addition, both random generation and optimal selection of the minimal set are performed in a weighted manner to make the rotation averaging pipeline more robust. Ablation studies and comparison experiments on the 1DSfM and San Francisco (SNF) datasets demonstrate the effectiveness of our proposed method.

Even though there have been a large amount of previous work on video segmentation techniques, it is still a challenging task to extract the video objects accurately without interactions, especially for those videos which contain irrelevant frames (frames containing no common targets). In this essay, a novel multivideo object cosegmentation method is raised to cosegment common or similar objects of relevant frames in different videos, which includes three steps: 1) object proposal generation and clustering within each video; 2) weighted graph construction and common objects selection; and 3) irrelevant frames detection and pixel-level segmentation refinement. We apply our method on challenging datasets and exhaustive comparison experiments demonstrate the effectiveness of the proposed method.

Co-segmentation addresses the problem of simultaneously extracting the common targets appeared in multiple images. Multiple common targets involved object co-segmentation problem, which is very common in reality, has been a new research hotspot recently. In this paper, an unsupervised object co-segmentation method for indefinite number of common targets is proposed. This method overcomes the inherent limitation of traditional proposal selection-based methods for multiple common targets involved images while retaining their original advantages for objects extracting. For each image, the proposed multi-search strategy extracts each target individually and an adaptive decision criterion is raised to give each candidate a reliable judgment automatically, i.e., target or non-target. The comparison experiments conducted on public data sets iCoseg, MSRC, and a more challenging data set Coseg-INCT demonstrate the superior performance of the proposed method.

In this paper, an object cosegmentation method based on shape conformability is proposed. Different from the previous object cosegmentation methods which are based on the region feature similarity of the common objects in image set, our proposed SaCoseg cosegmentation algorithm focuses on the shape consistency of the foreground objects in image set. In the proposed method, given an image set where the implied foreground objects may be varied in appearance but share similar shape structures, the implied common shape pattern in the image set can be automatically mined and regarded as the shape prior of those unsatisfactorily segmented images. The SaCoseg algorithm mainly consists of four steps: 1) the initial Grabcut segmentation; 2) the shape mapping by coherent point drift registration; 3) the common shape pattern discovery by affinity propagation clustering; and 4) the refinement by Grabcut with common shape constraint. To testify our proposed algorithm and establish a benchmark for future work, we built the CoShape data set to evaluate the shape-based cosegmentation. The experiments on CoShape data set and the comparison with some related cosegmentation algorithms demonstrate the good performance of the proposed SaCoseg algorithm.

For interactive segmentation approaches, object segmentation in complicated background is cumbersome, and usually needs tedious interactions to refine the incomplete segmentations . In this paper, an adaptive optimal shape prior is proposed for easy interactive object segmentation. Different from the traditional shape priors which only provide loose constraint, our adaptive shape prior gives more accurate and individualized constraint by exploiting the shape information of incomplete segmentation. Moreover, by combining the non-rigid shape registration and a local shape consistency evaluation system presented in this paper, such adaptive optimal shape prior could be achieved automatically. Both of these contributions greatly lighten the burden on users and make interactive segmentation much easier. The comparison experiments on the newly-built TypShape dataset with the related algorithms have demonstrated good performance of the proposed algorithm.