Domain Generalized Semantic Segmentation (DGSS) has recently attracted lots of research attention aiming to achieve robust segmentation performance on unseen domains, which aligns with diverse real-world applications. Visual Foundation Model (VFM), depending on large-scale pre-trained data and exquisite training strategies with good generalizability, has been explored and mined in some downstream tasks like DGSS. However, existing VFM-based DGSS methods predominantly focus on fine-tuning the VFM to adapt them to the semantic segmentation task. While this improves task alignment, it inevitably introduces additional training overhead and still results in static prediction behavior at inference time, making the model unable to adapt to varying target-domain distributions during deployment. To address these issues, we propose aTraining-free VFM-guided Dynamic Refinement (TVDR) frameworkfor the DGSS task, operating at the inference stage. First, a maximum category voting module is proposed to smooth the segmentation result of large masks generated by Segment Anything Model (SAM). Second, a small-object under-segmentation optimization strategy is proposed to improve the generalization of the small objects. Finally, a fusion refinement module is designed to refine the segmentation results of the above strategies. Compared with existing methods, our strategy does not require any parameter updates for VFM and has the advantages of plug-and-play and flexible deployment. It provides an efficient and practical new paradigm for cross-domain semantic segmentation tasks. Extensive experiments on widely-used benchmarks verified the effectiveness of the proposed approaches. Code is available at: https://github.com/Hectoor/TVDR.

The continuous breakthroughs in 3D reconstruction and rendering technologies have enabled synthesized 3D models to achieve exceptional realism. In particular, Neural Radiance Fields (NeRF) and 3D Gaussian Splattings (3DGS) have gained significant attention due to their impressive ability to deliver high-quality 3D models. However, research on the quality assessment of NeRF/3DGS models remains an underexplored area, hindering the development of relevant generation, compression, and transmission algorithms. To fill this gap, in this paper, we propose the first no-reference quality assessment metric for NeRF/3DGS models rendered in Processed Video Sequences (PVS). Considering the uniqueness and diversity of distortions introduced in NeRF/3DGS models, the core idea behind the proposed metric is to extract universal quality-aware features that are generalizable across various distortions, rather than targeting a specific one. Specifically, inspired by the fact that spatial distortions typically alter the statistical distributions, we first measure the spatial fidelity of the rendered PVS by analyzing the spatial explicit statistics of textural variation and naturalness. Then, motivated by the ability of implicit energy composition changes to reflect temporal distortions, we propose to evaluate the temporal consistency of the rendered PVS through inter-frame discrepancy energy and multi-frame motion energy in the Singular Value Decomposition (SVD) domain. Finally, the spatial explicit statistics and temporal implicit energy are combined as perceptual features to evaluate the quality of NeRF/3DGS models via Support Vector Regression (SVR). Extensive experimental results on three representative databases demonstrate the superiority of the proposed metric in various aspects, such as predictive accuracy, performance stability, and cross-database generalizability. The source code will be publicly available at https://github.com/ZhengyuZhang96/PVS-3DMQA.

With the rise of mobile computing and the increasing demand for real-time applications, the need for efficient and accurate semantic segmentation models has become paramount. However, existing state-of-the-art models are often hindered by heavy computational requirements, rendering them impractical for real-time applications. To tackle this challenge, we introduce the Contextual Guidance Network (CGNet), an efficient and lightweight network designed specifically for real-time semantic segmentation in autonomous driving. CGNet primarily consists of two key components: the Contextual Guidance Module (CGM) and the Triple-Branch Residual Fusion Module (TRFM). The CGM is comprised of the Downsampling Refine Unit (DRU) and the Contextual Guidance Bottleneck (CGB), which are utilized to gather dense contextual information. The DRU functions as a downsampling tool to generate low-resolution images, while the CGB extracts rich contextual information from both spatial and channel dimensions. Additionally, the TRFM utilizes the Residual Fusion Module (RFM) to achieve feature fusion and enhance pixel prediction accuracy. Without bells and whistles, CGNet achieves impressive mean intersection over union (mIoU) scores of 77.11% with 1.00 million parameters at 86.71 frames per second (fps) on the Cityscapes dataset, 72.26% mIoU at 88.62 fps on the CamVid dataset, and 63.32% mIoU on the BDD100K dataset. Extensive experiments demonstrate that CGNet achieves a favorable tradeoff between segmentation accuracy, inference speed and computational cost, making it suitable for autonomous driving systems with limited hardware resources. The source code will be available on GitHub: https://github.com/lv881314/CGNet

Existing domain generalization semantic segmentation (DGSS) methods have achieved remarkable performance on unseen domains by generating stylized images to increase the diversity of training data. However, since the training data is usually class-imbalanced, uniform style randomization is unable to generate diverse minority classes. This means that models may overfit to the minority classes, resulting in suboptimal performance on the minority classes. In addition, the image-level style randomization may also corrupt the class-discriminative regions of objects, leading to a loss of the class-discriminative representation. To address these issues, a novel class-balanced sampling and discriminative stylization (CSDS) approach is proposed for DGSS. Specifically, first, a pixel-level class-balanced sampling (PCS) strategy is proposed to adaptively sample patches of the minority classes from the source domain images and paste the sampled patches on the input images. Unlike existing class sampling strategies that fix the minority classes, the PCS strategy dynamically determines the minority classes by estimating the class distribution after each sampling. Then, a class-discriminative style randomization (CSR) strategy is proposed to increase the style diversity of the sampled patches while preserving the class-discriminative regions. Finally, since the pasting positions of the sampled patches are uncertain, which may confuse the semantic relations between the classes, a semantic consistency constraint is proposed to ensure the learning of reliable semantic relations. Extensive experiments demonstrate that the proposed approach achieves superior performance compared to existing DGSS methods on multiple benchmarks. The source code has been released onhttps://github.com/seabearlmx/CSDS.

Generalizable semantic segmentation, aims to excel on unseen target domains, as a critical focus due to the widespread practical applications requiring high generalizability. Class-wise prototypes, which depict class-wise centroids, as a type of domain-invariant information are key to improving the model generalizability due to its stability and representativeness. However, this manner faces some challenges. First, the existing methods adopt a coarse prototypical alignment form, potentially compromising performance. Second, the naive prototype generally serves as the class centroid generated by an average operation from source data batches, risks source domain overfitting, and may be detrimentally impacted by unrelated source data. Third, from a broader perspective, rather than just from a prototypical alignment perspective, the existing methods treat all samples equally, which is against the conclusion that different source features have different adaptation difficulties. To tackle these issues, we propose a novel method for generalizable semantic segmentation called Prototypical Progressive Alignment and Reweighting (PPAR) depending on the strong generalized representation of the Contrastive Language-Image Pretraining (CLIP) model. In particular, we first define the Original Text Prototype (OTP) and Visual Text Prototype (VTP) generated by the CLIP model, laying the foundation for the subsequent effective alignment strategy. Then, we propose a prototypical progressive alignment strategy by an easy-to-difficult alignment form to reduce domain-variant information progressively instead of directly. Finally, we propose a prototypical reweighting learning strategy that estimates the importance of the source data and corrects its learning weight to alleviate the influence of unrelated source features, i.e. alleviate negative transfer. Moreover, we also offer a theoretical insight into our method and it shows that our method compiles well on the domain generalization theory. Extensive experiments on several popular datasets demonstrate that our PPAR method achieves superior performance, proving the effectiveness of our method. The source code will available at: https://github.com/Hectoor/PPAR

Unsupervised domain adaptative semantic segmentation is a powerful solution for the distribution shift problem between the source and target domains. However, such methods need specified target domain data that may be unavailable in actual applications due to excess expensive collection. Generalizable semantic segmentation as a new paradigm appears in recent research, which aims to generalize well on distinct unseen domains only using source domain data. The existing methods focus on learning domain-invariant features by using global distribution alignment strategies, which may lead to a decreased discriminability of the model. To cope with this challenge, we propose a fine-grained self-supervision (FGSS) framework for generalizable semantic segmentation that takes into account both discriminability and generalizability from the perspective of the intra-class relationship. The FGSS framework contains single-view and multi-view versions. In the single-view version, we propose a fine-grained self-supervision strategy to distinguish the sub-parts of the semantic class for better class discriminability. In the multi-view version, we propose a class prototype feature enhancement strategy to generate another view (i.e. another representation of the original representation). Then, we propose a multi-view mutual supervision loss to enforce consistency between different views and further enhance the generalizability of the model. Experimental results on five widely-used datasets, i.e., GTAV, SYNTHIA, BDD100K, Cityscapes, and Mapillary, demonstrate that our FGSS framework achieves superior performance compared to state-of-the-art methods.

Unsupervised domain adaptation semantic segmentation (UDASS) methods aim to learn domain-invariant information for alleviating the distribution shift problem between the source and target domains. However, ignoring the learning of domain-specific information that is label-related may limit the class discriminability on the target domain. We argue that a good representation for the UDASS task not only contains domain-invariant information but also preserves label-related domain-specific information. In this paper, a novel frequency spectrum domain adaptation approach via meta-learning (ML-FSDA) is proposed to achieve this goal for improving the class discriminability and generalization ability. ML-FSDA contains a frequency-spectrum meta-learning framework (FMF) and a class-aware domain-specific memory bank (CDMB). Specifically, first, inspired by the observation that the high-frequency component is consistent across different domains while the low-frequency component is much more domain-specific, the FMF aims to respectively learn label-related domain-specific and domain-invariant information from low-frequency and high-frequency images in a unified framework via the meta-learning strategy. Second, the CDMB is designed to preserve the label-related domain-specific information of each class in an external memory bank while the CDMB is updated in every iteration of the meta-training stage. Finally, the CDMB is utilized to embed the label-related domain-specific information into domain-invariant information at the class level during the meta-testing stage to enhance the class discriminability on the target domain. Extensive experiments demonstrate the effectiveness of ML-FSDA on two challenging cross-domain semantic segmentation benchmarks. Notably, for the GTA5 to Cityscapes task and the SYNTHIA to Cityscapes task, the proposed ML-FSDA achieves superior performance with 77.3% mIoU and 68.8% mIoU, respectively. The source code is released at https://github.com/seabearlmx/FSL.

Prototypical contrastive learning (PCL) has been widely used to learn class-wise domain-invariant features for domain generalization semantic segmentation. These methods assume that the prototypes in different domains are invariant. However, the prototypes in different domains have discrepancies as well. First, the prototypes of the same class in different domains may be different. Second, the prototypes of different classes may be similar. To address these issues, a calibration-based multi-prototype contrastive learning (CMPCL) approach is proposed, which contains an uncertainty-guided multi-prototype contrastive learning (UMPCL) and a hard-weighted multi-prototype contrastive learning (HMPCL). Specifically, the UMPCL uses an uncertainty probability matrix, derived from element-wise discrepancies between the prototypes of the same class, to calibrate the weights of prototypes for alleviating the discrepancy between the prototypes of the same class in different domains. The HMPCL uses a hard-weighted matrix that is generated by the similarity between the prototypes of different classes, to calibrate the weights of the hard-aligned prototypes for alleviating the issue of similar prototypes between different classes, with hard-aligned prototypes referring to those exhibiting such similarity. Furthermore, since the learned class-wise domain-invariant features may overfit the prototype in the source domain, multi-prototype contrastive learning is used in the UMPCL and HMPCL to avoid this risk. Extensive experiments demonstrate that our approach achieves superior performance over current approaches on multiple benchmarks of domain generalization semantic segmentation. The source code has been released onhttps://github.com/seabearlmx/CMPCL.

Light Field Image Quality Assessment (LF-IQA) is vitally important to facilitate the development of immersive technologies. However, current state-of-the-art LF-IQA metrics still struggle to handle Light Field Image (LFI) with massive data in an efficient manner. To cope with this challenge, we propose a simple yet effective Blind LF-IQA metric based on Spatio-Angular Textural Variation, named SATV-BLiF. Given a distorted LFI, we first apply Local Binary Pattern (LBP) operator to measure the textural variation in the spatial and angular domains respectively. Then the generated spatial and angular textural matrices are merged and further transformed into statistical textural histogram features. Finally, Support Vector Regression (SVR) is employed to construct a nonlinear mapping function between the statistical textural histogram features and the perceptual quality score of the distorted LFI. Experimental results on three representative light field databases show that the proposed metric achieves state-of-the-art quality evaluation performance, while having much lower complexity than the existing No-Reference (NR) LF-IQA metrics. The code of the proposed SATV-BLiF metric is available at https://github.com/ZhengyuZhang96/SATV-BLiF.

Prototypical contrastive learning (PCL) has been widely used to learn class-wise domain-invariant features recently. These methods are based on the assumption that the prototypes, which are represented as the central value of the same class in a certain domain, are domain-invariant. Since the prototypes of different domains have discrepancies as well, the class-wise domain-invariant features learned from the source domain by PCL need to be aligned with the prototypes of other domains simultaneously. However, the prototypes of the same class in different domains may be different while the prototypes of different classes may be similar, which may affect the learning of class-wise domain-invariant features. Based on these observations, a calibration-based dual prototypical contrastive learning (CDPCL) approach is proposed to reduce the domain discrepancy between the learned class-wise features and the prototypes of different domains for domain generalization semantic segmentation. It contains an uncertainty-guided PCL (UPCL) and a hard-weighted PCL (HPCL). Since the domain discrepancies of the prototypes of different classes may be different, we propose an uncertainty probability matrix to represent the domain discrepancies of the prototypes of all the classes. The UPCL estimates the uncertainty probability matrix to calibrate the weights of the prototypes during the PCL. Moreover, considering that the prototypes of different classes may be similar in some circumstances, which means these prototypes are hard-aligned, the HPCL is proposed to generate a hard-weighted matrix to calibrate the weights of the hard-aligned prototypes during the PCL. Extensive experiments demonstrate that our approach achieves superior performance over current approaches on domain generalization segmentation tasks. The source code will be released at https://github.com/seabearlmx/CDPCL.

In recent years, convolutional neural networks have achieved significant success in computer vision tasks. However, the deployment of these algorithms remains challenging. Knowledge distillation (KD) as a type of important method enables a tiny model to extract helpful information from a large model. Most existing KD methods based semantic segmentation aim to align predicted maps in the spatial domain, but channel distillation also may help to improve segmentation performance. Additionally, pairwise pixel affinity provides efficiently structured reasoning for semantic segmentation. Motivated by these considerations, we propose a novel Channel Affinity KD (CAKD) framework for semantic segmentation that focuses on channel and cross-channel affinity relationship distillation to better align the distribution of the student and teacher models. Extensive experiments demonstrate that our proposed approach outperforms state-of-the-art KD methods on Cityscapes, Pascal VOC, and ADE20k datasets.

Road semantic segmentation is an essential component of autonomous driving and blind navigation. Although many excellent RGB-based road semantic segmentation algorithms have been proposed, these methods may not detect correctly due to the lack of geometric information. Recently, RGB-D road semantic segmentation methods attract more research attention. However, the existing RGB-D methods ignore the impact of unknown noise in sensors. To solve this problem, we propose an RGB-D Sequence Progressive Network (SPNet) for road semantic segmentation. Specifically, we first propose a sequence-based RGB-D feature extractor to alleviate the effect of noise. Then, We propose a multi-modal feature fusion (MMFF) module to enhance the feature representation of multi-modal data by further alleviating the effect of noise. Finally, we propose a semantic flow prediction (SFP) module that aims to align the multi-modal features in the decoder. Extensive experiments are conducted on several challenging datasets, including KITTI and GMRP. Our method achieves an F-score of 97.21% on the KITTI official leaderboard and ranked third in the official leaderboard.

Semantic segmentation assigns a category for each pixel and has achieved great success in a supervised manner. However, it fails to generalize well in new domains due to the domain gap. Domain adaptation is a popular way to solve this issue, but it needs target data and cannot handle unavailable domains. In domain generalization (DG), the model is trained without the target data and DG aims to generalize well in new unavailable domains. Recent works reveal that shape recognition is beneficial for generalization but still lack exploration in semantic segmentation. Meanwhile, the object shapes also exist a discrepancy in different domains, which is often ignored by the existing works. Thus, we propose a Shape-Invariant Learning (SIL) framework to focus on learning shape-invariant representation for better generalization. Specifically, we first define the structural edge, which considers both the object boundary and the inner structure of the object to provide more discrimination cues. Then, a shape perception learning strategy including a texture feature discrepancy reduction loss and a structural feature discrepancy enlargement loss is proposed to enhance the shape perception ability of the model by embedding the structural edge as a shape prior. Finally, we use shape deformation augmentation to generate samples with the same content and different shapes. Essentially, our SIL framework performs implicit shape distribution alignment at the domain-level to learn shape-invariant representation. Extensive experiments show that our SIL framework achieves state-of-the-art performance.

Multi-task learning is a successful learning framework which improves the performance of prediction models by leveraging knowledge among related tasks. Referring expression comprehension (REC) and segmentation (RES) are highly relevant tasks, which both are language-guided visual recognition tasks. However, their relations have not yet been fully exploited in previous works. In this paper, a Multiple Relational Learning Network (MRLN) is proposed for multi-task learning of REC and RES. First, a feature-feature interaction learning module is introduced to handle the complicated interactions among features. Moreover, we propose a feature-task dependence learning module, which associates the related features with target tasks. Furthermore, a task-task relationship learning module is designed, which captures the relationships among tasks automatically and guides the REC and RES fine-tuning adaptively. To verify our proposed approach, experiments are conducted on three benchmark datasets, i.e., RefCOCO, RefCOCO+, and RefCOCOg. Extensive experiments demonstrate that the multiple relationships are more appealing since it alleviates the prediction inconsistency issue in multi-task setup. In addition, the experimental results report the significant performance gains of MRLN over most existing methods, i.e., up to 83.46 % for REC and 63.62 % for RES over state-of-the-art methods, which demonstrate the validity and superiority of MRLN.