Synchronization is All You Need: Exocentric-to-Egocentric Transfer for Temporal Action Segmentation with Unlabeled Synchronized Video Pairs

We consider the problem of transferring a temporal action segmentation systeminitially designed for exocentric (fixed) cameras to an egocentric scenario,where wearable cameras capture video data. The conventional supervised approachrequires the collection and labeling of a new set of egocentric videos to adaptthe model, which is costly and time-consuming. Instead, we propose a novelmethodology which performs the adaptation leveraging existing labeledexocentric videos and a new set of unlabeled, synchronizedexocentric-egocentric video pairs, for which temporal action segmentationannotations do not need to be collected. We implement the proposed methodologywith an approach based on knowledge distillation, which we investigate both atthe feature and Temporal Action Segmentation model level. Experiments onAssembly101 and EgoExo4D demonstrate the effectiveness of the proposed methodagainst classic unsupervised domain adaptation and temporal alignmentapproaches. Without bells and whistles, our best model performs on par withsupervised approaches trained on labeled egocentric data, without ever seeing asingle egocentric label, achieving a +15.99 improvement in the edit score(28.59 vs 12.60) on the Assembly101 dataset compared to a baseline modeltrained solely on exocentric data. In similar settings, our method alsoimproves edit score by +3.32 on the challenging EgoExo4D benchmark.