Current Convolutional Neural Networks (CNNs) for Weakly Supervised Semantic Segmentation (WSSS) often have difficulties in discovering distinctive feature locations for each category. Therefore, the pseudo-labels generated from the expanded seed regions are typically incomplete and contain a significant amount of noise. Without additional annotations, the numerous erroneous information will potentially propagate in the segmentation network’s training stage. In this work, we propose a Cross-Modal Dual Graph Reasoning (CDGR) framework to leverage both visual and language knowledge effectively. This framework can capture dependencies between the spatial and the semantic spaces, facilitating the discovery of discriminative feature locations. Specifically, we perform cross-modal graph reasoning between the visual and the language modal graphs to enhance global contextual relationships between pixels in the visual feature map. Additionally, we introduce a graph interaction attention network to thoroughly explore implicit relationships between visual and language graphs. We apply the CDGR network to generate more complete pseudo-labels for the classification network and utilize it in the segmentation network to unleash its self-correcting capabilities. Extensive experiments on the PASCAL VOC 2012 and MS COCO 2014 datasets demonstrate the effectiveness of CDGR compared to other state-of-the-art peers. Our code is provided at https://github.com/JIA-ZHANG666/CDGR.

In recent years, deep learning-based superpixel segmentation methods derived from SLIC have made significant progress by utilizing uniform grid-based seed initialization. However, due to the unequal pixel space variation rates in natural images, methods based on uniform grid initialization struggle to balance the compactness of superpixels in flat regions with the boundary adherence in non-flat regions. Inspired by the visual attention model based on saliency in the human visual system, we propose a content-aware dynamic superpixel segmentation network. Specifically, we propose a seed initialization strategy guided by geodesic distance transformation and design two segmentation heads for different scales, which are used for joint network training to encourage the network to focus more on areas with texture variations without causing unnecessary segmentation in flat regions. Extensive experiments on BSDS500 and NYUv2 datasets demonstrate that our method achieves state-of-the-art performance.

Establishing global contextual relationships between objects is crucial for weakly supervised semantic segmentation (WSSS) tasks that lack pixel-level labels. Limited by the efficiency of convolutional operations in capturing long-range dependencies with a limited receptive field and to bridge the gap between image-level annotations and pixel-level labels, we propose a Dual Graph Reasoning Mapping (DGRM) module. When integrated into a convolutional network, it conducts contextual graph reasoning on both spatial and interaction spaces of visual features. The first component of this graph reasoning module involves incorporating commonsense knowledge extracted from an external knowledge base into visual features to promote global contextual reasoning for visual graphs. The second component focuses on reasoning in the projected interaction space, utilizing abstracted object class attributes from high-level visual features to establish dependencies among channels in a potential low-dimensional space. Moreover, to capture correspondences at different semantic levels, we model the feature maps in a pyramid-like structure for graph reasoning at various levels. Extensive experiments on popular datasets, such as PASCAL VOC 2012 and MS COCO 2014, demonstrate the superiority of our approach. Our code is provided athttps://github.com/JIA-ZHANG666/DGRM.

Pavement crack detection is a practical and challenging task that has the ability to significantly reduce the burden of manual building and road maintenance in intelligent transportation systems. Existing methods mainly focus on addressing common crack diseases and are poor in generalizing to other conditions of crack detection due to diverse environmental factors (e.g., illumination), topology complexity, and intensity in-homogeneity. Moreover, the samples suffer from the severe foreground-background imbalance and the model is easily prone to overfitting on trained anomalies, resulting in unsatisfactory performance. To tackle the aforementioned challenges and achieve high-quality pavement crack detection, we propose an innovative approach termed boundary-aware axial attention network (BAAN), which is composed of multiple position-guided axial attention (PAA) modules in a hierarchical encoder-decoder architecture. Specifically, it learns efficient contextual information via decomposed multidimensional position-guided attention to capture more precise spatial structures, and the proposed boundary regularization module (BRM) mines more discriminative foreground-background relationships to regularize the ambiguous details between diverse spatial regions. Moreover, we propose a novel boundary refinement loss (BRL) to alleviate the challenges associated with regional losses (e.g., pixel-wise cross-entropy loss) in the context of heavily imbalanced crack detection problems. The proposed BAAN is evaluated on four crack datasets and experimental results indicate that the BAAN consistently outperforms the state-of-the-art methods with fewer computational requirements.

Abstract Automatic skin lesion segmentation is the most critical and relevant task in computer‐aided skin cancer diagnosis. Methods based on convolutional neural networks (CNNs) are mainly used in current skin lesion segmentation. The requirement of huge pixel‐level labels is a significant obstacle to achieve semantic segmentation of skin lesion by CNNs. In this paper, a novel weakly supervised framework for skin lesion segmentation is presented, which generates high‐quality pixel‐level annotations and optimizes the segmentation network. A hierarchical image segmentation algorithm can predict a boundary map for training images. Then, the optimal regions of candidate hierarchical levels are selected. Afterward, Superpixels‐CRF built on the optimal regions is guided by spot seeds to propagate information from spot seeds to unlabeled regions, resulting in high‐quality pixel‐level annotations. Using these high‐quality pixel‐level annotations, a segmentation network can be trained and segmentation masks can be predicted. To iteratively optimize the segmentation network, the predicted segmentation masks are refined and the segmentation network are retrained. Comparative experiments demonstrate that the proposed segmentation framework reduces the gap between weakly and fully supervised skin lesion segmentation methods, and achieves state‐of‐the‐art performance while reducing human labeling efforts.

The performance of salient object detection (SOD) has been significantly advanced by using deep convolutional networks. However, it largely depends on the high cost of pixel-level annotations. To reduce human effort while improving the prediction accuracy, we propose a novel two-phase learning framework. The weakly supervised information in terms of scribbles is provided as initial labels. Then, as the first phase, high-quality pseudo-labels are generated by mapping scribbles onto object/object-part contours. These contour maps are predicted by the hierarchical contour detection algorithm, providing superior accuracy and smoothness. In the second phase, a deep neural network is alternately trained and predicted. The pseudo-labels are refined in an iterated process, where a conditional random field (CRF) model and a filter module are designed to promote the performance. Extensive experiments on five benchmarks show that our framework can achieve comparable results with the state-of-the-art fully and weakly supervised methods.

Abstract Hierarchical image segmentation is a prevalent technique in the literature for improving segmentation quality, where the segmentation result needs to be searched at different scales of the hierarchy to identify objects represented from various scales. In this paper, a novel framework for improving the quality of object segmentation is presented. To this end, the authors first select the optimal segments among several hierarchical scales of the input image using simple mid‐level features and dynamic programming. Simultaneously, deep seeds are localised on the input image for the foreground and background classes using a deep classification network and a saliency network, respectively. Then, a graphical model is constructed as a set of nodes that jointly propagate information from deep seeds to unmarked regions to obtain the final object segmentation. Comprehensive experiments are performed on different datasets for popular hierarchical image segmentation algorithms. The experimental results show that the proposed framework can significantly improve the quality of object segmentation at low computational costs and without training any segmentation network.

In this paper, we present a road crack segmentation method based on generative adversarial networks (GAN). Our GAN networks consist of two neural network models in terms of a generator and a discriminator, where two improved networks CU-Net and FU-Net are proposed based on U-Net. The U-Net, CU-Net and FU-Net are used as the generator, while two-class networks are used as the discriminator. The purpose of using the generator is to generate fake crack images which are very similar to real crack images. And the recognition is done by the discriminator to distinguish the real crack images from the fake crack images. After iterative training between the generator and the discriminator, the generator can generate fake crack images that are very similar to the real crack image. Finally, the generator can be used to segment the road crack images. Compared with the other state-of-the-art methods on three datasets, the proposed method achieves better performance. Specifically, the precision, recall and F1-score are 91.46%, 73.40%, and 77.33%, respectively on one of the public datasets.

Evaluating image segmentation quality is a critical step for generating desirable segmented output and comparing performance of algorithms, among others. However, automatic evaluation of segmented results is inherently challenging since image segmentation is an ill-posed problem. This paper presents a framework to evaluate segmentation quality using multiple labeled segmentations which are considered as references. For a segmentation to be evaluated, we adaptively compose a reference segmentation using multiple labeled segmentations, which locally matches the input segments while preserving structural consistency. The quality of a given segmentation is then measured by its distance to the composed reference. A new dataset of 200 images, where each one has 6 to 15 labeled segmentations, is developed for performance evaluation of image segmentation. Furthermore, to quantitatively compare the proposed segmentation evaluation algorithm with the state-of-the-art methods, a benchmark segmentation evaluation dataset is proposed. Extensive experiments are carried out to validate the proposed segmentation evaluation framework.

Quantitative evaluation of image segmentation quality is usually based on comparing a segmentation with multiple reference segmentations. Instead of holistically comparing with each reference, we propose a region based evaluation framework, where an exemplar reference is adaptively constructed and applied to a generally defined evaluation measure. As examples, we implement three well-known evaluation measures and present an efficient scheme to compute each measure. Extensive experiments on the benchmark databases show that the proposed evaluation framework can improve the evaluation precision of existing measures.

In this letter, we propose a probabilistic measure to evaluate the machine segmentation with multiple ground truths. The measure is designed for adaptively evaluating the structural information extracted from the segmentations. This induces a local similarity score at every point in the segmentation and can in turn be accumulated in a principled information-theoretic way into a global similarity score of the entire segmentation. Experiments are conducted on benchmark images from the Berkeley segmentation database and our own database. Results show that the proposed measure can faithfully reflect the perceptual qualities of the segmentations.

This paper addresses the automatic image segmentation problem in a region merging style. With an initially oversegmented image, in which many regions (or superpixels) with homogeneous color are detected, an image segmentation is performed by iteratively merging the regions according to a statistical test. There are two essential issues in a region-merging algorithm: order of merging and the stopping criterion. In the proposed algorithm, these two issues are solved by a novel predicate, which is defined by the sequential probability ratio test and the minimal cost criterion. Starting from an oversegmented image, neighboring regions are progressively merged if there is an evidence for merging according to this predicate. We show that the merging order follows the principle of dynamic programming. This formulates the image segmentation as an inference problem, where the final segmentation is established based on the observed image. We also prove that the produced segmentation satisfies certain global properties. In addition, a faster algorithm is developed to accelerate the region-merging process, which maintains a nearest neighbor graph in each iteration. Experiments on real natural images are conducted to demonstrate the performance of the proposed dynamic region-merging algorithm.