Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training

Contrastive learning has emerged as a promising paradigm for 3D open-worldunderstanding, i.e., aligning point cloud representation to image and textembedding space individually. In this paper, we introduce MixCon3D, a simpleyet effective method aiming to sculpt holistic 3D representation in contrastivelanguage-image-3D pre-training. In contrast to point cloud only, we develop the3D object-level representation from complementary perspectives, e.g.,multi-view rendered images with the point cloud. Then, MixCon3D performslanguage-3D contrastive learning, comprehensively depicting real-world 3Dobjects and bolstering text alignment. Additionally, we pioneer the firstthorough investigation of various training recipes for the 3D contrastivelearning paradigm, building a solid baseline with improved performance.Extensive experiments conducted on three representative benchmarks reveal thatour method significantly improves over the baseline, surpassing the previousstate-of-the-art performance on the challenging 1,156-category Objaverse-LVISdataset by 5.7as text-to-3D retrieval and point cloud captioning, further evidencing itsefficacy in diverse scenarios. The code is available athttps://github.com/UCSC-VLAA/MixCon3D.