PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails

Large language models (LLMs) are typically aligned to be harmless to humans.Unfortunately, recent work has shown that such models are susceptible toautomated jailbreak attacks that induce them to generate harmful content. Morerecent LLMs often incorporate an additional layer of defense, a Guard Model,which is a second LLM that is designed to check and moderate the outputresponse of the primary LLM. Our key contribution is to show a novel attackstrategy, PRP, that is successful against several open-source (e.g., Llama 2)and closed-source (e.g., GPT 3.5) implementations of Guard Models. PRPleverages a two step prefix-based attack that operates by (a) constructing auniversal adversarial prefix for the Guard Model, and (b) propagating thisprefix to the response. We find that this procedure is effective acrossmultiple threat models, including ones in which the adversary has no access tothe Guard Model at all. Our work suggests that further advances are required ondefenses and Guard Models before they can be considered effective.