In the domain of computer vision, Parameter-Efficient Tuning (PET) is increasingly replacing the traditional paradigm of pre-training followed by full fine-tuning. PET is particularly favored for its effectiveness in large foundation models, as it streamlines transfer learning costs and optimizes hardware utilization. However, the current PET methods are mainly designed for single-modal optimization. While some pioneering studies have undertaken preliminary explorations, they still remain at the level of aligned encoders (e.g., CLIP) and lack exploration of misaligned encoders. These methods show sub-optimal performance with misaligned encoders, as they fail to effectively align the multimodal features during fine-tuning. In this paper, we introduce DETRIS, a parameter-efficient tuning framework designed to enhance low-rank visual feature propagation by establishing dense interconnections between each layer and all preceding layers, which enables effective cross-modal feature interaction and adaptation to misaligned encoders. We also suggest using text adapters to improve textual features. Our simple yet efficient approach greatly surpasses state-of-the-art methods with 0.9% to 1.8% backbone parameter updates, evaluated on challenging benchmarks.

In this paper, we propose TGS-LGP, a framework for text-guided medical image segmentation that synergistically combines cross-modal understanding with local-global features perception. Our method employs a local-global text encoder that simultaneously processes clinical language inputs through two complementary pathways: one leverages contrastive language-image pretraining to capture global semantic context, and another employes domain-specific linguistic analysis to extract fine-grained lesion characteristics. These multiscale textual representations are progressively fused with refined visual features through an attention-based decoder that dynamically fuse cross-modal features. This architectural design enables context-aware integration of diagnostic text guidance with radiological image patterns. Additionally, we introduce vision refiner, a novel component that effectively integrates global features of medical images with minimal computational cost, compensating for the information loss caused by the local modeling bias of CNNs in medical imaging. This enhancement strengthens visual representation, thereby facilitating more effective fusion with textual features. Experimental results on two widely used benchmarks demonstrate that our method achieves the best accuracy compared to state-of-the-art methods.

Leveraging multimodal large models for image segmentation has become a prominent research direction. However, existing approaches typically rely heavily on manually annotated datasets that include explicit reasoning processes, which are costly and time-consuming to produce. Recent advances suggest that reinforcement learning (RL) can endow large models with reasoning capabilities without requiring such reasoning-annotated data. In this paper, we propose SAM-R1, a novel framework that enables multimodal large models to perform fine-grained reasoning in image understanding tasks. Our approach is the first to incorporate fine-grained segmentation settings during the training of multimodal reasoning models. By integrating task-specific, fine-grained rewards with a tailored optimization objective, we further enhance the model's reasoning and segmentation alignment. We also leverage the Segment Anything Model (SAM) as a strong and flexible reward provider to guide the learning process. With only 3k training samples, SAM-R1 achieves strong performance across multiple benchmarks, demonstrating the effectiveness of reinforcement learning in equipping multimodal models with segmentation-oriented reasoning capabilities.