LLaNA: Large Language and NeRF Assistant

Multimodal Large Language Models (MLLMs) have demonstrated an excellentunderstanding of images and 3D data. However, both modalities have shortcomingsin holistically capturing the appearance and geometry of objects. Meanwhile,Neural Radiance Fields (NeRFs), which encode information within the weights ofa simple Multi-Layer Perceptron (MLP), have emerged as an increasinglywidespread modality that simultaneously encodes the geometry and photorealisticappearance of objects. This paper investigates the feasibility andeffectiveness of ingesting NeRF into MLLM. We create LLaNA, the firstgeneral-purpose NeRF-language assistant capable of performing new tasks such asNeRF captioning and Q&A. Notably, our method directly processes the weights ofthe NeRF's MLP to extract information about the represented objects without theneed to render images or materialize 3D data structures. Moreover, we build adataset of NeRFs with text annotations for various NeRF-language tasks with nohuman intervention. Based on this dataset, we develop a benchmark to evaluatethe NeRF understanding capability of our method. Results show that processingNeRF weights performs favourably against extracting 2D or 3D representationsfrom NeRFs.