InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction

Text-conditioned human motion generation has experienced significantadvancements with diffusion models trained on extensive motion capture data andcorresponding textual annotations. However, extending such success to 3Ddynamic human-object interaction (HOI) generation faces notable challenges,primarily due to the lack of large-scale interaction data and comprehensivedescriptions that align with these interactions. This paper takes theinitiative and showcases the potential of generating human-object interactionswithout direct training on text-interaction pair data. Our key insight inachieving this is that interaction semantics and dynamics can be decoupled.Being unable to learn interaction semantics through supervised training, weinstead leverage pre-trained large models, synergizing knowledge from a largelanguage model and a text-to-motion model. While such knowledge offershigh-level control over interaction semantics, it cannot grasp the intricaciesof low-level interaction dynamics. To overcome this issue, we further introducea world model designed to comprehend simple physics, modeling how human actionsinfluence object motion. By integrating these components, our novel framework,InterDreamer, is able to generate text-aligned 3D HOI sequences in a zero-shotmanner. We apply InterDreamer to the BEHAVE and CHAIRS datasets, and ourcomprehensive experimental analysis demonstrates its capability to generaterealistic and coherent interaction sequences that seamlessly align with thetext directives.