Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing

Deep Text-to-Image Synthesis (TIS) models such as Stable Diffusion haverecently gained significant popularity for creative Text-to-image generation.Yet, for domain-specific scenarios, tuning-free Text-guided Image Editing (TIE)is of greater importance for application developers, which modify objects orobject properties in images by manipulating feature components in attentionlayers during the generation process. However, little is known about whatsemantic meanings these attention layers have learned and which parts of theattention maps contribute to the success of image editing. In this paper, weconduct an in-depth probing analysis and demonstrate that cross-attention mapsin Stable Diffusion often contain object attribution information that canresult in editing failures. In contrast, self-attention maps play a crucialrole in preserving the geometric and shape details of the source image duringthe transformation to the target image. Our analysis offers valuable insightsinto understanding cross and self-attention maps in diffusion models. Moreover,based on our findings, we simplify popular image editing methods and propose amore straightforward yet more stable and efficient tuning-free procedure thatonly modifies self-attention maps of the specified attention layers during thedenoising process. Experimental results show that our simplified methodconsistently surpasses the performance of popular approaches on multipledatasets.