Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models

Diffusion Models (DMs) have exhibited superior performance in generatinghigh-quality and diverse images. However, this exceptional performance comes atthe cost of expensive architectural design, particularly due to the attentionmodule heavily used in leading models. Existing works mainly adopt a retrainingprocess to enhance DM efficiency. This is computationally expensive and notvery scalable. To this end, we introduce the Attention-driven Training-freeEfficient Diffusion Model (AT-EDM) framework that leverages attention maps toperform run-time pruning of redundant tokens, without the need for anyretraining. Specifically, for single-denoising-step pruning, we develop a novelranking algorithm, Generalized Weighted Page Rank (G-WPR), to identifyredundant tokens, and a similarity-based recovery method to restore tokens forthe convolution operation. In addition, we propose a Denoising-Steps-AwarePruning (DSAP) approach to adjust the pruning budget across different denoisingtimesteps for better generation quality. Extensive evaluations show that AT-EDMperforms favorably against prior art in terms of efficiency (e.g., 38.8saving and up to 1.53x speed-up over Stable Diffusion XL) while maintainingnearly the same FID and CLIP scores as the full model. Project webpage:https://atedm.github.io.