Large Multimodal Models (LMMs) have demonstrated remarkable capabilities in multimodal understanding and conversation. Recently, some researchers have explored fine-tuning LMM for pixel grounding, leading to catastrophic loss of their inherent conversational capabilities. To preserve the conversational capability, some researchers explore to freeze LMM, but employ heavy segmenter SAM for high-quality grounding. In this paper, we propose a novel approach, named HG-LMM, to fully exploit the inherent features of frozen LMM for high-quality pixel grounding. Our HG-LMM introduces two main modules: an LLM-guided instance-aware feature generation (LIFG) and a layer-wise detail and semantic injection (LDSI). The LIFG module employs the output text embeddings of LMM belonging to grounded instances to generate multi-level instance-aware feature maps from the image encoder. Afterwards, we employ the LDSI module to inject more detail and semantic information into these instance-aware feature maps. With these instance-aware feature maps, we employ a simple top-down fusion to predict the segmentation masks of different instances. We perform experiments on various tasks, including referring expression segmentation, panoramic narrative grounding, reasoning segmentation, grounded conversation generation, and visual chain-of-thought reasoning. When using DeepSeekVL-1.3B, our HG-LMM is 12.6% better than F-LMM without SAM in terms of segmentation accuracy on the all set of PNG dataset. Compared to F-LMM with SAM, our HG-LMM achieves comparable segmentation accuracy while being 2.7 times faster. We release our source code and models at https://github.com/WenjieLi2008/HG-LMM.

Current super-resolution (SR) algorithms rely heavily on annotated data and often ignore the uncertainty in image degradation and features, limiting their real-world application. We propose an uncertainty-aware SR network using a self-convolutional attention mechanism. Our approach focuses on an SR reconstruction network enhanced by a cross-scale self-convolutional attention mechanism within the Transformer framework, which leverages local regions at various resolutions as convolution kernels to enhance high-frequency details. We also design a heteroscedastic uncertainty loss function to learn pixel and feature uncertainties, guiding the network to improve textures and edges adaptively. Extensive experiments show that our method achieves superior visual reconstruction on standard real-world datasets.

Haze significantly affects image clarity, hindering vision-based systems like autonomous driving. However, current dehazing methods rely heavily on labeled data and overlook pixel-level uncertainties, limiting real-world effectiveness. To address this, we propose an uncertainty-driven weakly supervised dehazing network with a regional dynamic attention module and a transformer-based multi-scale feature fusion module. Firstly, the regional dynamic attention module employs a channel-pixel collaborative attention mechanism to adjust weights based on haze distribution, enhancing focus on hazy channels and dense regions. Meanwhile, the multi-scale feature fusion module integrates global semantics with local details through window-based multi-head self-attention and parallel dilated convolutions, effectively capturing long-range dependencies while preserving details. Additionally, we introduce an unsupervised loss function based on heteroscedastic uncertainty to guide training by adaptively correcting contrast, enhancing edges, and improving saturation. Extensive experiments on synthetic and real-world images demonstrate that our network shows superior performance in both objective quality and subjective evaluation.

Text-to-image diffusion models have shown powerful ability on conditional image synthesis. With large-scale vision-language pre-training, diffusion models are able to generate high-quality images with rich textures and reasonable structures under different text prompts. However, adapting pre-trained diffusion models for visual perception is an open problem. In this paper, we propose an implicit and explicit language guidance framework for diffusion-based visual perception, named IEDP. Our IEDP comprises an implicit language guidance branch and an explicit language guidance branch. The implicit branch employs a frozen CLIP image encoder to directly generate implicit text embeddings that are fed to the diffusion model without explicit text prompts. The explicit branch uses the ground-truth labels of corresponding images as text prompts to condition feature extraction in diffusion model. During training, we jointly train the diffusion model by sharing the model weights of these two branches. As a result, the implicit and explicit branches can jointly guide feature learning. During inference, we employ only implicit branch for final prediction, which does not require any ground-truth labels. Experiments are performed on two typical perception tasks, including semantic segmentation and depth estimation. Our IEDP achieves promising performance on both tasks. For semantic segmentation, our IEDP has the mIoU$^\text{ss}$score of 55.9% on ADE20K validation set, which outperforms the baseline method VPD by 2.2%. For depth estimation, our IEDP outperforms the baseline method VPD with a relative gain of 11.0%.

Weakly supervised Referring Expression Grounding (REG) aims to localize the target entity in an image based on a given expression, where the mapping between image regions and expressions is unknown during training. It faces two primary challenges. Firstly, conventional methods involve selecting regions to generate reconstructed texts for computing the backpropagation loss between regions and expressions. However, semantic deviations in text reconstruction may result in significant cross-modal bias, leading to substantial losses even in cases of correctly matched regions. Secondly, the absence of region-level ground truth in weakly supervised REG results in a lack of stable and reliable supervision during training. To tackle these challenges, we propose a Progressive Semantic Reconstruction Network (PSRN), which utilizes a two-level matching-reconstruction process based on the key triad and adaptive phrases, respectively. We leverage progressive semantic reconstruction with a three-staged training strategy to mitigate the deviations in the reconstructed texts. Additionally, we introduce a Constrained Interactions operation and an Attention Coordination mechanism to facilitate additional bidirectional supervision between the two matching processes. Experiments on three benchmark datasets of RefCOCO, RefCOCO+ and RefCOCOg demonstrate that the proposed PSRN has the competing results. Our source code will be released athttps://github.com/5jiahe/psrn.

Recently, deep learning-based single image dehazing method has been a popular approach to tackle dehazing. However, the existing dehazing approaches are performed directly on the original hazy image, which easily results in image blurring and noise amplifying. To address this issue, the paper proposes a DPDP-Net (Dual-Path in Dual-Path network) framework by employing a hierarchical dual path network. Specifically, the first-level dual-path network consists of a Dehazing Network and a Denoising Network, where the Dehazing Network is responsible for haze removal in the structural layer, and the Denoising Network deals with noise in the textural layer, respectively. And the second-level dual-path network lies in the Dehazing Network, which has an AL-Net (Atmospheric Light Network) and a TM-Net (Transmission Map Network), respectively. Concretely, the AL-Net aims to train the non-uniform atmospheric light, while the TM-Net aims to train the transmission map that reflects the visibility of the image. The final dehazing image is obtained by nonlinearly fusing the output of the Denoising Network and the Dehazing Network. Extensive experiments demonstrate that our proposed DPDP-Net achieves competitive performance against the state-of-the-art methods on both synthetic and real-world images.