Connecting NeRFs, Images, and Text

Neural Radiance Fields (NeRFs) have emerged as a standard framework forrepresenting 3D scenes and objects, introducing a novel data type forinformation exchange and storage. Concurrently, significant progress has beenmade in multimodal representation learning for text and image data. This paperexplores a novel research direction that aims to connect the NeRF modality withother modalities, similar to established methodologies for images and text. Tothis end, we propose a simple framework that exploits pre-trained models forNeRF representations alongside multimodal models for text and image processing.Our framework learns a bidirectional mapping between NeRF embeddings and thoseobtained from corresponding images and text. This mapping unlocks several noveland useful applications, including NeRF zero-shot classification and NeRFretrieval from images or text.