Recent text-to-image diffusion models have shown significant advancements in high-quality image synthesis. Represented by Stable Diffusion, these models have been applied in various fields, such as art design, digital development, personalization, etc. However, they still fail to generate images that properly comply with the given text prompts, especially when encountering multiple subjects. Current approaches for multi-subject synthesis focus on latent code optimization at specific timesteps,yet yielding compromised text-image alignment. In this paper, we explore the generation of diffusion models through the lens of both noise initialization and denoising process, and propose an effective two-stage Attention Refinement and Inheritance (ARI) approach. Firstly, we update the initial noise by refining the cross-attention maps between different subject tokens, employing strategies of separation, energization, and centralization. Then, we introduce attention-level interpolation as an inheritancemechanism to explicitly regulate the denoising process. This enables the image generation to maintain high correspondence with the refined initial attention, thereby achieving a more controllable synthesis. Extensive experiments show that our approach achieves more faithful alignment between generated images and text prompts compared with state-of-the-art multi-subject image synthesis methods.

The remarkable development of text-to-image generation models has raised notable security concerns, such as the infringement of portrait rights and the generation of inappropriate content. Concept erasure has been proposed to remove the model’s knowledge about protected or inappropriate concepts. Although many methods have tried to balance the efficacy (erasing target concepts) and specificity (retaining irrelevant concepts), they can still generate abundant erasure concepts under the steering of semantically related inputs. In this work, we propose Corer to address this "concept residue" issue. Specifically, we first introduce the mechanism of neighbor-concept mining to dig out the associated concepts and expand the erasing range. Furthermore, to mitigate the negative impact on the generation of irrelevant concepts caused by the expansion of erasure scope, Corer preserves the specificity through the beyond-concept regularization. We also employ the closed-form solution to optimize weights of U-Net, as well as the prediction noise alignment with the LoRA module. Extensive experiments on multiple benchmarks demonstrate that Corer outperforms previous concept-erasing methods in terms of superior erasing efficacy, specificity, and generality.

Image captioning is a crucial task that bridges computer vision and natural language processing, with applications ranging from assistive technologies to content creation. Recently, retrieval-augmented generation (RAG) methods have been introduced to enhance captioning models by incorporating external knowledge and enabling training-free domain transfer. These approaches utilize pre-trained retrievers to fetch relevant information for input images, providing additional context to the caption generation process. However, we observe that high-similarity retrieved results do not always enhance the generation process, revealing a misalignment between retrieval and generation in existing methods. To address this issue, we propose Retrieval Alignment for Caption Enhancement (RACE), a novel RAG-based image captioning method. Instead of merely enhancing the similarity between the input image and the retrieved results, our RACE method focuses on retrieving results that more effectively benefit the final generation outcomes. Specifically, the retriever is optimized to align with the image captioning model. We introduce a generation-oriented similarity correction loss to align the similarity-based distribution with the CIDEr-based evaluation distribution of the generated captions. In this proposed loss function, greater emphasis is placed on training samples with higher divergence. This ensures that the retriever retrieves more relevant results preferred by the language decoder, while preserving pre-trained knowledge. Extensive experiments on both in-domain (COCO) and out-of-domain (NoCaps, Flickr30k, VizWiz) benchmarks demonstrate that RACE outperforms other RAG-based image captioning methods. By effectively bridging the gap between retrieval and generation, RACE enhances overall captioning quality.

Recently, adversarial examples are introduced to protect personal images from being identified by unauthorized face recognition systems. Existing approaches follow the transfer-based adversarial attack paradigm, where local surrogate models are utilized to generate protected images. However, these surrogate models can neither be necessary nor efficient for generating adversarial examples. In this paper, we propose SD4Privacy, i.e., Stable Diffusion for Privacy, which exploits the latent space of Stable Diffusion Model to synthesize adversarial examples. First, we learn an optimal textual embedding of target image to preserve its representative semantics, directly guiding the sampling process of synthesized image. Then, we utilize the encoder of UNet in Stable Diffusion as the substitution of surrogate classification models, which enables the efficient adversarial guidance by semantic h-space of UNet for adversarial example generation. Experiments show the state-of-the-art protection performance, as well as high-quality protected images with visual naturalness and imperceptible perturbations.

With the development of diffusion-based customization methods like DreamBooth, individuals now have access to train the models that can generate their personalized images. Despite the convenience, malicious users have misused these techniques to create fake images, thereby triggering a privacy security crisis. In light of this, proactive adversarial attacks are proposed to protect users against customization. The adversarial examples are trained to distort the customization model's outputs and thus block the misuse. In this paper, we propose DisDiff (Disrupting Diffusion), a novel adversarial attack method to disrupt the diffusion model outputs. We first delve into the intrinsic image-text relationships, well-known as cross-attention, and empirically find that the subject-identifier token plays an important role in guiding image generation. Thus, we propose the Cross-Attention Erasure module to explicitly "erase" the indicated attention maps and disrupt the text guidance. Besides, we analyze the influence of the sampling process of the diffusion model on Projected Gradient Descent (PGD) attack and introduce a novel Merit Sampling Scheduler to adaptively modulate the perturbation updating amplitude in a step-aware manner. Our DisDiff outperforms the state-of-the-art methods by 12.75% of FDFR scores and 7.25% of ISM scores across two facial benchmarks and two commonly used prompts on average.