Universal Cross-Domain Retrieval (UCDR) aims to retrieve images across unseen domains and categories, a critical capability for real-world applications. While large-scale Vision-Language Models (VLMs) like CLIP offer strong zero-shot category generalization, they struggle with domain shifts. Existing methods often improve domain robustness at the cost of high computational overhead or by compromising the VLM's inherent knowledge. To address this, we propose Decoupled and Fused Tuning with LoRA (DeFT-LoRA), a novel and parameter-efficient framework that integrates Low-Rank Adaptation (LoRA) with a Mixture-of-Experts (MoE) mechanism. This approach resolves the intrinsic conflict between domain-invariant and domain-specific knowledge in a single adapter, enabling our model to construct a domain adapters for each input image. We propose a three-stage training strategy, which first learns a shared Base LoRA for domain-invariant features, then derives Domain-Specific Experts to capture specific styles, and finally fuses them dynamically with a lightweight gating network. Extensive experiments on three UCDR benchmarks demonstrate that DeFT-LoRA achieves comparable or superior performance to state-of-the-art methods while requiring only 1.46 percent of CLIP's image-encoder parameters and reducing computational overhead, thereby establishing an exceptional balance between accuracy and efficiency.

Low-Rank Adaptation (LoRA) is a widely utilized technique in topic of Parameter-Efficient Transfer Learning (PETL) which could use a limited number of trainable parameters to adapt the model to various downstream tasks. However, the setting of the locations and low-rank sizes in traditional LoRA relies heavily on the fixed and empirical values, which may hinder adaptability and lead to sharply decreasing performance, especially on some self-supervised pre-trained models. To alleviate this dilemma, we introduce a feature responsive LoRA (ResLoRA) method, a resource-efficient algorithm that automatically determines the LoRA modules’ required size based on the downstream task’s response. Firstly, we propose a Feature Decomposition loss (FD-loss) which leverages the feature singular values to mine the corresponding features of different downstream tasks, making the model parameters able to adequately represent downstream tasks. Subsequently, we leverage the Taylor expansion to measure the salience of the model parameters, then some high-efficient parameters with high significance could be leveraged to design a dynamically responsive LoRA. Specifically, the location and low-rank sizes of LoRA are determined based on the response parameters of the features for downstream tasks. Extensive experiments show that our ResLoRA achieves state-of-the-art performance, especially in the transfer capability of self-supervised models based on MoCo v3 and MAE. Our code is available at: https://github.com/wildboarman/ResLoRA.