SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos

We propose a novel self-supervised embedding to learn how actions sound fromnarrated in-the-wild egocentric videos. Whereas existing methods rely oncurated data with known audio-visual correspondence, our multimodalcontrastive-consensus coding (MC3) embedding reinforces the associationsbetween audio, language, and vision when all modality pairs agree, whilediminishing those associations when any one pair does not. We show our approachcan successfully discover how the long tail of human actions sound fromegocentric video, outperforming an array of recent multimodal embeddingtechniques on two datasets (Ego4D and EPIC-Sounds) and multiple cross-modaltasks.