SM^3: Self-Supervised Multi-task Modeling with Multi-view 2D Images for Articulated Objects

Reconstructing real-world objects and estimating their movable jointstructures are pivotal technologies within the field of robotics. Previousresearch has predominantly focused on supervised approaches, relying onextensively annotated datasets to model articulated objects within limitedcategories. However, this approach falls short of effectively addressing thediversity present in the real world. To tackle this issue, we propose aself-supervised interaction perception method, referred to as SM^3, whichleverages multi-view RGB images captured before and after interaction to modelarticulated objects, identify the movable parts, and infer the parameters oftheir rotating joints. By constructing 3D geometries and textures from thecaptured 2D images, SM^3 achieves integrated optimization of movable part andjoint parameters during the reconstruction process, obviating the need forannotations. Furthermore, we introduce the MMArt dataset, an extension ofPartNet-Mobility, encompassing multi-view and multi-modal data of articulatedobjects spanning diverse categories. Evaluations demonstrate that SM^3surpasses existing benchmarks across various categories and objects, while itsadaptability in real-world scenarios has been thoroughly validated.