DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation

World models have demonstrated superiority in autonomous driving,particularly in the generation of multi-view driving videos. However,significant challenges still exist in generating customized driving videos. Inthis paper, we propose DriveDreamer-2, which builds upon the framework ofDriveDreamer and incorporates a Large Language Model (LLM) to generateuser-defined driving videos. Specifically, an LLM interface is initiallyincorporated to convert a user's query into agent trajectories. Subsequently, aHDMap, adhering to traffic regulations, is generated based on the trajectories.Ultimately, we propose the Unified Multi-View Model to enhance temporal andspatial coherence in the generated driving videos. DriveDreamer-2 is the firstworld model to generate customized driving videos, it can generate uncommondriving videos (e.g., vehicles abruptly cut in) in a user-friendly manner.Besides, experimental results demonstrate that the generated videos enhance thetraining of driving perception methods (e.g., 3D detection and tracking).Furthermore, video generation quality of DriveDreamer-2 surpasses otherstate-of-the-art methods, showcasing FID and FVD scores of 11.2 and 55.7,representing relative improvements of 30