Genie: Generative Interactive Environments

We introduce Genie, the first generative interactive environment trained inan unsupervised manner from unlabelled Internet videos. The model can beprompted to generate an endless variety of action-controllable virtual worldsdescribed through text, synthetic images, photographs, and even sketches. At11B parameters, Genie can be considered a foundation world model. It iscomprised of a spatiotemporal video tokenizer, an autoregressive dynamicsmodel, and a simple and scalable latent action model. Genie enables users toact in the generated environments on a frame-by-frame basis despite trainingwithout any ground-truth action labels or other domain-specific requirementstypically found in the world model literature. Further the resulting learnedlatent action space facilitates training agents to imitate behaviors fromunseen videos, opening the path for training generalist agents of the future.